Decades back, Dan Henry said in relation to this issue "bicycling is an anti-social sport". I think he meant non-social, rather than anti-social, but the point is spot on.

Well matched riders may ride in a tight peloton, but mixed groups will have each rider ride at his own pace, and that's fine.

Arrange breaks with the understanding that some early birds may leave before slowpokes arrive, but accept that socializing on the ride will be in small groups of matched riders, or friends that opt to stay together. If you want a social aspect, plan on a social event at the end starting early enough and lasting long enough for everyone to partake.

Any other attempt at keeping the group together on the ride will alienate more riders than it pleases.