I would wager as both a "roadie" and a fixed rider that you're probably passing the same roadies that Im passing on my road bike.
There are LOTS of people who buy expensive ass bikes... wear the "uniform" just to go for like a 10 mile ride. Trust me. I used to work at a shop. I guess everyone has a different way of "living".
Not me though. I'll be the guy doing the passing thank you very much.