What's the problem? Are you worried that people will question your sexuality? Do you think they'll make your butt look fat?
I don't understand this aversion to clothing made specifically for the thing you do on a regular (if not daily) basis. I don't generally wear stuff like that when I go to class or work, but if I was racing or on a century you better ****ing believe I'll have that **** on. It really does make you more comfortable because it's supposed to.
You could always just take a pair with you and change at the track too.