I ride a bicycle, and I drive a car too. There aren't many cars that come within three feet of my bike. I stay as far as possible to the right, most cars are five, seven, even ten feet or more to my left. I think someone's been "buzzing" the cyclists, which is the reason we have this hostility.
These group rides are composed of cyclists with many skill levels, from expert to novice. Some of them are acting as "human road cones" for the rest of the group. And the men are going to protect the women, which is an important fact to consider, before you go berzerk.
But remember, you have a car; if a large group of cyclists is blocking the road, you can turn a different direction at the next intersection, and detour-take an alternate route to your destination.
I try to sympathize with motorists, because I drive too, but I have found that motorists accuse cyclists of blocking the road even when the bikes are doing the speed limit. Which is the end of the argument. We started off listening to your complaint, assuming that the bikes were going really slow, and then it turns out they were really fast.