I just bought a 2004 ES330 and i'm very happy so far. I too researched a lot of different models and the ES transmission issue and i felt comfortable enough to spend my money on the Lexus.
I drove an 02 and 03 ES300 and i could get the transmission to be a little "confused" around 40mph, but overall the car(s) were smooth. I drove the 04 and from all my test driving and the miles i've logged on my car so far, it's been nothing but great.
Since i drove a few different cars (Lexus and others) I will say that every car was different as far as the exceleration / tranmissions go. I've had to get used to the new ES and can drive it very smooth. I'm not sure what to make of all the transmission fuss because if you floor it all the time, it does 'hesitate'. But, in my opinion, if you want a smooth car and don't want to race everywhere, the ES330 is perfect.
Like the advice mentioned before, take a drive and buy what you like. Like any other car, it take some getting used to, but i've been really impressed with the car overall.
I based my decision on the quality of car, smooth ride, quite, decent stereo and gas milage, lots of standard features, safety, niiiiice inside, etc, so it's been perfect for me so far.