Hi lizzy,
I drive an '06 ES330 with 95k on the ODO. The one thing you may want to look into (on both cars) is if the timing belt was changed. There stands a higher probability that the '05 had it done (Lexus recommends at 90k) and that the '04 hasn't since it's only 5k beyond that and people like to stretch things sometimes. If they both had it done, that's a major expense you can cross off. Pricing is similar but the biggest differences between those two are the fact that the '05 was the refresh for the generation.
Changes included (may not be listing them all):
Revisied taillights
Revised headlights (projector beans and round fogs)
Standard 17" inch rims over the 16's.
Dual memory seats
New steering wheel with much better intergrated controls
To me, there is a big difference in the looks between the '04 and '05 models, def. a more luxury look, and in my case I would be choosing the '05. As gbhrps said, make sure you get it checked out, and if they were properly maintained, they'll go a lot longer.