As a 50+'er if you're not used to guys (and gals) blowing by you as if you were standing still, well you better get used to it!
All we can do is be happy with our own fitness and progress. Compare yourself to your colleagues at work, not the riders you meet on the road! You're probably in the top 10% of fitness among all age groups at work no?