hemi roadster

I am trying to punch out a small dent in the side of our RX350, and it is in the same spot as the cargo speaker. I figured the speaker grill or rim would pop off with some reasonable pressure, but no luck so far. Before I destroy something, does anyone know of the proper removal method? Once the speaker is out, I figure I will have access to the body panel, otherwise I will have to go in from underneath. Thanks from a new member.