I suppose too that it depends on what you call a fast group ride.
I've never been on any group ride of any sort that didn't have to stop at intersections and such. All but the slowest riders in my club can catch up at a stop sign waiting to cross a state highway. I'm in New York State, after all, and there's a lot of traffic even in "rural" areas. Plus, cycling three miles without a stop sign is cause for celebration. (And it explains why I find the MUP such an attraction that I commute on it daily. There's a five-mile stretch with no stops.)
In any event, I've never been on any group ride of any sort that didn't stop at least once an hour to regroup, snack, or have a pee.
In those circumstances, three minutes per hour isn't a hill of beans--or even a small pile. It's 5%, or the difference between a 20MPH average and a 19MPH average.
Put another way, 5% is the approximate difference between each of the gears on my cassette, in the 12-13-14-15-16-17-18-19 range (I ride a ten-speed 12-23.) One shift. Not a huge difference.
Maybe I just ride with the "wrong" groups. Or have the wrong idea of what constitutes a huge difference in speed.
I still maintain that if you put Chris Froome on my bike, yes, he'd lose the jersey, but he'd still make the time cut. Me? I'd never even make the time cut on his bike.
I think it depends on whether you're that last guy to catch up at a stop sign as the group takes off again (after they had short respite) while you get to keep going.
On some rides with technically the same group, I've been that guy, or one of those guys. I deliberately chose to ride with a group that would really push me. On other days I can easily keep up. It all depends who shows up and how I'm feeling.
There's more than one day where I'd have really loved to be able to add 1 mph to my average speed without having to work any harder. To me that's a pretty significant difference in that context even though it's not at all significant while commuting. On a 40 mile group ride, someone averaging 20 mph vs 19 will be two miles ahead when they finish. They're going to be enjoying their waffles before I even get back to the shop.
I do agree that you're not going to jump from a B rider to an A rider by going from a Cross Check to a performance road bike. But if you're a marginal A/B rider with a Crosscheck on a hilly route, you might fit a little more comfortably into the A/B category with a lighter bike.