Hi Matt, I own the 08 es 350 and my wife owns and loves the 08 IS 250 awd. I drive hers enough to know the pros and cons compared to one another. I like the quiter ride, the roomier interior size, and the trunk room when we travel 1500 miles each way to key west. Trips that long would not be as comfortable in the IS. I'm 5'10, 195 and the IS is a little cramped for me. She is 5'4, 125 and loves small cars like her previous Carolla. I notice the tire/road noise in her car and the stiffer suspension. If someone is in the back seat the seat belt recepticals jab that person in the hip area. It drives great, looks awsome and the brakes are the best I've ever experienced. That being said, I still would take the ES over the IS because comfort is much more important to me than looks. Although I am 69 and my wife is 65 I still would have felt this way in my 30's. As to how to reply to the people who pose that question, suggest that they experience driving them both (at the dealer, not your car), and then tell you which one they would choose.
Once doing so they may understand why you chose the ES. One thing is for sure, they are both great cars. JIm B