The more you ride, the faster you'll get and the more you'll at least pass the stragglers. I once caught up to a group ride of roadies while commuting on my hybrid complete with stuffed panniers. I asked if I could draft a bit (strong headwind that day as well). When I told them was 12 miles into my 17 mile commute, they were just "No WAI!" I rode with them a mile or so, then actually passed them. They weren't an advanced group obviously, but I'd be lying if I said it didn't feel good.