If you go through life worried sick about what other people think, or if you let stupid comments bother you, then you might as well buy a stationary bike and stay inside.
I am 58 years old, stand 6'3" tall and weigh 240 lbs. I wear lycra bib shorts, wild, bright jerseys and ride in broad daylight. I could not care less what my neighbors think. I love to ride and the clothing I choose make it more comfortable and safer.