That is a good idea, riding with your same age group/fitness. I am not fortunate. In Thailand (I am a Thai), there are very few roadbike riders. So I have to ride with younger guys (30-40) regularly. I got dropped only when I started cycling 8-9 years ago. Over the years, my fitness improves and now I can ride comfortably with young guns, never get dropped and sometimes even in the front group.
Keep riding and gradually extend your distance. One day, you will be surprised that your 55-years-old body (mine) can keep up with young guns easily.