That's why I never progressed beyond intermediate level in amateur races in my 20s. I could ride all day with little rest, even centuries, as long as it was my own pace. But I couldn't adjust to the ebb and flo of the pack, and my endurance would be sapped by pushing only slightly harder than my comfort zone.
Nowadays I don't even worry about it. The only groups I ride with are casual city rides. And I only push my self when I'm riding alone, about once a week on the nearest thing we have to a hilly route. I can't think of a single time I've passed anyone who was riding a road bike locally -- I've tried pacing off them from behind but can never keep up for even a mile. Although I often pass other riders on hybrids and city bikes. Those city bike riders make my 12 mph loafing pace feel like I'm flying.