I have a 2004 GX which has been almost trouble free - the only problem was a latch on the rear door that failed and had to be replaced. Otherwise my experience with this car has been very good. I can imagine your feelings about the problems you mentioned.
I looked at the LX, too, when I was looking at the GX. It ended up being a cost decision - the new price between the models was significant to me. The things I liked better about the LX are:
1) The 2-piece rear door, with a liftagte/tailgate (instead of the single large door on the GX)
2) There is more storage room behind the 3rd row seat on the LX. Not a huge difference, but enough to stow a few duffels. On the GX I have to put 1/2 the 3rd seat up, which reduces the room for teen-agers sitting back there
3) I think there is a little more 3rd row leg room?
The things I liked worse on the LX:
1) It seems to have more things that require maintenance, and more often, than the GX. The fewer trips to the dealer, it seems the happier I am
2) A more complex AWD system than the GX, since I don't do any extreme off-road driving anyway
A couple of other things to think about:
1) Was your problem with the GX related more to the car, or to the dealer's ability to fix a problem? I know several people with GXs, and none of us have had the problems with driveline that are described here. There's nothing worse, car-wise, than the expectation that comes from owning a Lexus, and having a dealer that is unable to get a problem corrected. Be careful you don't just trade cars if the problem is really the dealer..
2) Make sure the LX has the options you want. My 04 GX has the Nav system and backup camera, which is a major deal for me because I have a neck injury, and it's a lot easier to back up with the camera. I REALLY like the audio system in the GX as well - make sure you're happy with the LX options, there may be some differences between the 03 & 04.
3) make sure the maintenance on the LX is up to date. You don't want to have a hidden cost to do a lot of maintenance, and the LX seems to have a few more items on the list.
Good luck with the change
Bob