I say go for it, Jon2016. It's actually fairly easy. Just go slow and use a small putty knife as IVAces suggests. I wish I would have had one at the time, but instead used a flat-head screw driver and left some small scratches that probably wouldn't be there had I used a small putty knife. Good luck!
Just wanted to say thanks for the detailed instructions on this thread. My work badge got stuck in the trap door of the center console on my wife's 2007 GX 470. While doing an online search on how to address it, I came across this thread. Followed the directions and had it all fixed and back together in about an hour, including time needed to tend to a crying toddler. Figured I'd sign up on here to say thanks as my first post! Sorry if bringing up this old thread is frowned upon.