Things like this are why I'm a huge fan of Strava flybys (though of course it only works if both you and the person that passed you are on Strava and don't have privacy locked down).

One day I got passed by this guy in a sleeveless jersey with no socks and aerobars. Huh?! I got home, checked Strava, and discovered that he was about 8 miles into a 20 mile ride with about 200 feet elevation gain, whereas I was about 8 miles into a 50 mile ride with about 2500 feet of elevation gain. He took a break around mile 10, I did not. Different training goals.

I've also found that around my parts, there are a lot of tough gals that can take down most guys that are not trained racers. The fastest women are not as fast as the fastest men, but can and do hang on A-group rides without getting dropped. Some of them are very competitive and will happily pass whatever moving target they encounter out on the road.