Stick with it man, and wear whatever you want. There will always be jerks in the world; if you were 165 pounds and wearing spandex, they'd probably be calling you ****** or something. Some people just need to push other people down to feel good about themselves. To heck with them.

If they don't like how you look, nothing's gluing their eyes to your ass.