When you get into 3 or 4 year old cars, it's the mileage that really counts, and of course, the previous maintenance.
I bought my 2002, two years ago, (it was one year old at the time) and it had all the Lexus service records and updates. I've got 36,000 miles on it now and it's good as new. The I6 engine, properly maintained, will run for 200,000 miles quite easily. Besides, from an appearance standpoint, a 2001 and a 2005 don't really look any different. Mileage and condition are the biggies.