I'm guessing you're lighter than they are, or you have a lot more miles in the bank. Do you like long mountainous rides? Take them up to Mt Palomar. If you beat them up Torrey Pines, you'll have to wait for them on a long climb.
My friend Valleygirl is the same age as you and she drops me and most other club riders in the mountains. She is right at the 2 pounds per inch of height mark. I don't think age is a big factor here.