Riding with the older guys/girls just means you get dropped by older people instead of younger people, so that's no solution. You might enjoy the company more in the meantime, though.

Working harder might work if that's what you want to do.

Right at the moment, I'm 50 and getting better- couldn't get worse!

Enjoying the ride even when you do get dropped is a bit of mental adjustment, and probably easier when you come at your level of ability from the lower side rather than the higher side.

I'm not sure why randonneuring and long-distance riding attracts the age group it does, but I think one major part is that it helps if you don't have kids at home. So it's not necessarily a physical thing.