Finally removed the door panel, thanks again for the help.
Here are the updated instruction for the 2005 es330:
First remove the door latch bezel. This is the plastic cup behind the door latch. Pry out this piece, pushing it rearward and lifting the tongue/latch after it's loose to pull the bezel forward and off. Remove the screw once you get the bezel out.
Next, using a screwdriver....carefully pop the electric door lock/window control center out (the whole black piece that has the electric buttons for the windows and door locks). Remove the screw behind the controls.
Next, start pulling out on the bottom of the door panel...pull pretty hard it will pull away from the door. Once it is pulled away from the door, lift up and it should come right off. (you will have several places to disconnect if you need the door panel completely removed (electric control center, cables for the locks and door handles, etc...).