I passed a young fellow in triathlon kit on the way home one night, but I don't fool myself that I'm fitter and faster. He looked exhausted and had probably just blown himself apart on a training ride. Also, many people use parts of the pathway system to get to the edge of town for highway rides and are very conscious of the need to ride carefully at a conservative speed to avoid collisions with pedestrians. Some people will blow past them and no doubt brag to their friends later that "I passed some bike racer poseur in lycra!"
One thing that irritates me is when I'm stopped at a red light or stop sign and someone will push past me to get in front because they just know they're going to be faster than me... and sometimes they aren't. So I end up on their back wheel looking for a safe way to pass when we get moving again. Or, they'll even ride straight through the intersection just to "win". Years ago I saw a young fellow nearly get hit by a garbage truck at a four-way stop, he was that desperate to get ahead of me. And I still ended up having to pass him further down the road. Last summer a fellow ran a red light to pass me... same again. I'd really like to know what it is about me that certain people feel compelled to compete with me. Why don't they find someone their own age and gender to "race" with???