I have been driving 2005 ES330 for 2 months. This car has this problem also. The hesitation is really annoying. Especially, when I enter the freeway, the car will not accelerate quick enough. I think this is danger and a faulty design.
My problem was that I didn't do enough research before I buy this car. This made me :( .
I dropped off my car for repair this morning (Dealer scartched my car when they repaired my seatbelt) and I got a loaner (2004 ES330). This loaner car is much better than my 2005 ES330 because it did not have the hesitation problem.
The good thing is that I am not alone. The bad thing is I am one of the victim. I will talk to the service consultant when I get my car back later. I hope he can fix it.