The gender of the passing rider is only important for aesthetic purposes.
Age is not a specific limitation either. Go watch the 65+ racers and be shocked at what old folks can do.
I knew a guy who said until he hit about 78 he never lost a beat. After that his recovery times went up ... he was 82 and faster than I'll ever be.
You are only on the downslope if you want to be.
On the other hand, I have no illusions about how fast I am ... or how motivated some riders are, simply to always go faster. I ride for pleasure, not pain ... so I am not as fast.