Some thoughts: I have noticed that people in cars tend to do the same things mentioned here, even though I bet less than 1% of motorists understand drafting. Herd instinct.
If on a street, a group is always more visible to others than one lone cyclist, so safety just might be increased a bit.
Road rage is probably more likely to be directed at a lone individual.
Assaults and such crimes are more likely to be directed at a lone individual; bullying is included here, so a group gets lets attention from bad guys.
And, for those who do not welcome attention from the badge guys, in my opinion, many negative encounters with police officers happen because the officer is really a bully with a badge, or someone who is still trying to adjust to his authority. (Some never learn.) As with the above thought on bad guys, there is safety in numbers. (I say this as someone who wears a badge, BTW.)
So, OK, drop the drafters, or, use them to your advantage.