I'm mostly even with "them". I ride about as long as anyone else but then I don't have a large group to ride I do find myself testing my past riding against the accumulated data from my GPS. Sometimes I race myself.
I don't know that I had a prime when it comes to riding a bicycle. I never competed, I just liked to ride which is how it is now. I ride off road more now, and find myself plenty capable of hanging with others on the same trails be they younger or older than me but to be honest I don't look at the hills and trails with the idea of beating anyone, I just like to conquer them. I use them as a cardio workout and I admit I dig the rush of a steep uphill, and the decent mixed with rocks, roots, and sometimes just air.
I'm an avid motorcyclist too with some racing and lots of track experience. I never had a problem keeping up there either and in fast surprised lots of guys in the pits when I took my helmet off.
I'm 55, 230-240 lbs depending and 6'1". I've been hitting the gym very regularly for the past 4 yrs and sporadically before that. I don't thank any of this makes me any more special or different than the next guy. I work out so I can do stuff like ride bikes, hike, camp, drink scotch and smoke a cigar once in awhile.