I've had that problem on both of my Lexus ('98 ES300 and '02 RX300) and the dealer replaced the blower motor on each. They tried to blame a clogged air filter, loose fittings, etc... but I have acute hearing and heard a high pitched squeal right at at the same tone as my tinnitus...
I had to moan and groan a couple of times, but it worked and motors were replaced.