You can try the repair, No guarantees on the sound though. I can get you a OEM replacement if needed.
If I get an OEM part, how can I remove the old one? Do I need to remove the side panel to get to the subwoofer?
Every speaker/sub cover on lexus have to be taken off from behind, So yes you have to remove the panel to get to the sub.