Don't worry about what people think about you. Ride and have fun. If your really worried about what people think don't worry about what your wearing or riding. Train your butt off and get faster, people can't judge people they can't catch. Also nothing would tick off a "hater" more than getting passed or dropped by a guy who they deem a fred.
Ride, Ride...Ride.