OP, ride your bike, wear what you want, enjoy the ride and for goodness sake stop worrying about what others wear, ride or think.
The only person whose thoughts concern me is the maniac behind the steering wheel of a two ton vehicle, texting his gf (or bf) and not paying attention to the road...that's the only person whose thoughts, or lack thereof, concerns me.
Cycling is a wonderfully simple activity...you...the bike...the road.
For the life of me I can't understand all the hand wringing about what gear others use/ride or what kit they wear.
Ride, be safe, enjoy!