I was told the same thing and I contacted them via Twitter. The 08 and 09 did have issues with the pearl white and they've addressed it because, like you mentioned, there were numerous calls. I told Lexus that a certified Lexus auto body shop and an insurance adjuster both stated it was not normal and that each vehicle should be uniquely addressed. Mine is on the roof and I can't tell you the last time I looked at my roof, I can't even see it (too short lol). They also stated that the service department at the dealership never mentioned it, well they don't inspect the roof.
If you have an 2004 Lexus GX470 and the paint is peeling, please take the time to call Lexus and submit a request for a goodwill claim. I would also use social media to convey this issue to other owners. I'm very disappointed with Lexus and if they don't resolve this, I certainly would not buy from them again.
