If your focus is on the clothing other cyclists are wearing, you're not riding hard enough.
btw, wrt wearing cycling kit, I keep hearing how silly people look all kitted out in pro team colors including gloves, shoes, helmet etc. Then I hear that it's ok if you ride for that team as you've earned it. Let's face it, to the average Joe out there we look stupid the second we put on any kit, be it all black or Garmin team colors. And you'll look equally dumb all kitted out in your local team crap. Don't believe me? Show up to the next Yankee game wearing your office team baseball gear, right down to the socks, cleats, helmet etc. That's exactly how you look on your bike in your local team gear to the non-cyclists out there. And who cares? Wear what you want.