Stop getting mad at your car!
Maybe someone else will chime in here, but in my experience any time you need to get into the center console of a Lexus. Toyota or even Honda/Acura requires removing an amazing amount of stuff. If I were you I would look for a radio install guide for your car and I'll bet that the removal instructions will be included in that.
Actually, my radar detector fell off the windshield, hit the switch and pushed it through! at least it didn't crack the wood!