Sound quality is one of the most subjective issues you can discuss. If it sounds good to you, then leave it alone. I've listened to a good number of the audio systems in the ES models, and strangely enough, have always found the stock sytem to sound better than the ML. With the 350 (07-08), the ML will play louder without distorting than stock, but at the sweet spot of each (clearest sound, best dynamic range and stereo separation), the stock system noses it out. That being said, they are both sub-par to what I would expect to hear in a lexus. The bass is almost non-existent in both (even discounting my preference for louder bass), and their sweet spot is in a very narrow range of volume. My 97 ES sounds much better than either of these newer offerings, (also with a narrow sweet spot, but wider than these two), and it's the stock radio. The new stock radio gets ok around half volume with a strong dr disc, and peters out around 2/3 volume, at which point the upper mids 3-5khz start screeching (not distorting, just too loud). The ML also gets better around half volume, and can crank up louder before the mid-highs 9-12khz begin doing the same. The problem could be somewhat related to the actual drivers and placements, but a much bigger problem lies in the head units themselves. Lexus engineers (as is the case with most car manufacturers) designed these head units to appeal to a very large target group, and built a sound system that is more unlikely to aggravate many, while less likely to please most audiophiles, (a much smaller target crowd). To that end, stereo separation is sacrificed at the lower volumes in favor of lower noise levels when playing the radio or tape, and dynamic range is cut at those same levels to promote a 'smoother sound'. Anybody claiming a flat frequency response across the volume spectrum in any stock radio needs to fix their RTA, scope, or whatever other piece of equipment they used to measure. None of them are flat, not one.
There are a few pieces of aftermarket equipment that deal specifically with this issue, while allowing you to retain your stock head unit, but be warned. Most require aftermarket amplifiers, and most require a decent understanding of audiophile equipment if you plan to self-install.
To get back to the original question posed in this thread, yes you should replace the sub if you aren't getting enough bass. You can try replacing the driver itself, but that will only give you an incremental improvement at best since you are dealing with an infinite baffle behind a rather small driver. A more convincing solution will involve a box with a pair of 10 or 12 inch subs and an amp to drive them. Hope this helps.