A couple of years ago, I was riding with a friend (who is 70+), we were doing a long charity ride, and were on the 4th day of a 5 day 450 mile ride - we average about 14mph. Anyway, we were cruising along, chatting about young women and whatnot, when a pair of younger riders passed us on our left. I started to pick up the pace a little to stay with the two young guys when my friend told me 'relax, when you are passed by someone half your age - that is how it should be'.
I don't worry too much about my speed, but I do take comfort in the fact that there are very few kids half my age who could do the number of miles that I do (I average 150-200 miles per week)