You can get away with anything as long as you're in shape and fast. Look at this guy. Can you imagine anyone else wearing spandex and a baggy sleeveless jersey?
Don't worry about how you look at an MTB "event". Just don't crawl down the singletrack at 5mph and block everyone behind you.