I've been riding for a long time, wearing anything but a proper cycling outfit would seem out of place.

But as long as it gets you to ride a bike or do some type of exercise, wear whatever you like.

ps. I wonder if swimmers get this type of question. hehe.