"Eat to please thyself, but dress to please others.” - Benjamin Franklin
Everyone should have a normal amount of modesty... as well as the self-confidence that promotes their own distinctive sense of style. But often times the refusal to wear the conventionally accepted fashions is just a form or reverse snobbery. Or someone trying to draw attention to themselves by standing out in unconventional outfits (hence, old Ben's quote).
There isn't anything "wrong" with cycling gear! Wearing the clothing/equipment/gear/costume (whatever you want to call it) is all part of the enjoyment of being part of a hobby/sport. Join in with the rest of us and have some fun. Plus... cycling specific gear really does have beneficial functions.
All that said/posted. Wear whatever you like! You're no less of a man, woman, or a cyclist just because you want to be different. There is a little bit of Fred in all of us.