If you are still in the decision making process then let me chime in. I own a 2000 RX300 and my father has an ML500. I have had plenty of behind the wheel time on the ML, here are my thoughts. These are totally different vehicles. The MLs are truck based vehicles while the RX is a car based vehicle. In fact, many ML owners throw some sort of fit when they are referred to as SUVs instead of truck (call it what it want).
But from a driver's standpoint, the ML is definitely a "rougher" car with clear truck roots. The ride is choppier (not bad, just choppier). The most important distinction in my view is that the ML demands more attention to drive whereas a RX is much easier to drive. Whether or not you think the amount of driving attentiveness is driving enjoyment or a lack thereof leads to a vehicle without road connection is for you to decide.
They are both nice cars, but I would have to say that the RX has better fit and finish and the 2004 model is the way to go.