+1 to all above. I'm 64 and not fast, but I've been riding for many years, and regularly drop guys in their 20s or 30s on their first group rides. 1 year later, 90% of them drop me...
And as someone said, there'll always be someone faster than you and someone slower too, so don't worry too much about it.