I finally removed my roof rails after 3 years of wanting to do so. This is for a 2007 RX350. Here are some pictures and a procedure for those who may want to do the same.
1. At the rear of the roof rack driver side, grab the rear black covering, lift slightly and pull towards the rear of the car. With some encouragement it will come off.
2. Do the same for the passenger side.
3. Remove the bolt in the pathway that would prevent removal of the rails. Removal of the yellow plastic piece is not necessary.
4. Slide the racks off.
5. Install the rear black covering by sliding it forward. You may have to press or bang the rear of it with your hand to get it all way in.
6. Done!