I thought that I was the only one with this problem. I also was hearing a rolling noise during acceleration and coming to a stop. I had to take my 04 RX330 in for service and I mentioned it to the service advisor. After keeping it for one week they finally found the problem.
According to the tech notes they heard the noise coming from the center floor area. They removed both front seats and the front carpet. There is a hollow portion in the center floor where they found a loose bolt. They removed it with a magnet, and the noise is gone. It seems that this bolt, which I don't know why it was loose, was rolling back and forth in the narrow hollow trough in the floor. I also don't know if they installed another one, and tightened it down, or if it was just bonus parts during the manufacturing process.
The bad news is they took apart my dash during their search and now it squeaks like crazy. Looks like another trip back to the dealer.