I assume you did a drain and refill. The A541E transmission holds 7.1 quarts of ATF (specification from 97 ES300 manual). Due to the transmissions design, on a drain and refill, only a portion of the transmission fluid drains out. In your case 3.7 quarts. So now you have about 50% new and 50% old ATF. That's why it's brown. Do a drain and refill two or three more times over a period of several weeks and as the percentage of new to old rises your fluid will become red.
There is a technique for replacing all the fluid. It's pretty easy. Search the Toyota site for instruction.