I used to have this theory that, as a reasonably good athlete in HS, I would eventually rise to an even higher comparative level as I aged --as long as I stayed fit. This was my "attrition" theory. In fact, the opposite has happened. Attrition seems to have knocked off the lower levels and left me at a lower level of survivors who are more fit and clearly better than me. Oops.
Regarding getting passed on a ride, I sometimes get some false respect because I ride a single speed (more theory?). The only hills around here are pretty gentle. I frequently pass people who have geared down to spin up the hill. I'm stuck at 70 gear-inches and have to power up them. As long as I spin my brains out on the straights for a while The riders I pass think I'm better than I really am.