First of all, I'm not too fast on my road bike yet. I was out on a rather flat MUP yesterday averaging 22 kph for a ~2hr ride. I had a couple of hybrids pass me, but not for a sustained effort. I caught them all eventually. Conclusion: it's easy for most people to do high speeds over short distances, so it doesn't bother me if I get passed. I have personal goals related to my average speed maintained over a certain amount of time and am happy to improve!

City riding is a different cup of tea. In the city I go at very moderate speeds, I don't filter, I stop for lights, etc. so getting passed constantly is par for the course, doesn't matter what bike I'm riding. It's super annoying because all these filterers and light jumpers pass me, and then I get stuck behind them when it is safe to get up to speed again.