salmoncheese,
That intermittent statement you left out of your first post suggests another issue that was present in that generation ES.
There is a relay in the engine compartment fuse box labelled "magnetic clutch" (I believe) that turns the AC compressor on and off. When it went bad, your interior fan still blew air, but the AC compressor wouldn't turn to move the refrigerant = no cooling. It usually made the dash AC button light flash repeatedly. If your relay (less than $20 to buy) is cutting in and out that could explain your problem. There is an updated part number for that relay meaning the newer one fixes the issue the old part had and shouldn't ever fail in the future.
You still could have an issue with a temperature sensor or a faulty mixing door actuator, but I'd start with my first two suggestions ... and have an AC expert (not necessarily a Lexus one) check the system out.
Gene