Hello JP: we had a simliar transmission problem on our 2004 RX330 with only about 50,000 miles on it at the time. Not only would ours hesitate on acceleration but would shift hard (thump) when moving to the next higher gear under a smoother part throttle. The hard shift did not occur under harder (closer to full throttle) acceleration.
On the first two trips to the dealer, he recalibrated the ECM twice (at no cost) as mentioned in one of the other replies. He said the problem was a result of Lexus moving to a complete drive-by-wire throttle system in the RX330 and asked that we drive it a month or more each time because the newly programmed ECM takes a while to learn the habits of the individual driver(s)and store them in its memory. That did not fix the issue, did not even phase it as a matter of fact, and the third time we took it in for the same problem, we told him to keep it as long as needed because it was unacceptable as it was and if it could not be fixed we would be getting rid of the vehicle. He kept it a over a week and replaced the transaxle under warranty (7 years, 70,000 miles drivetrain). Problem solved. Maybe this reply comes too late if you have already replaced the vehicle.