I do weekend rides with a group of 4 or 5 guys. On any given weekend some are inevitably faster and ride up front or get away altogether. The good thing about having more than just 2 is usually we end up paired up and are able to ride with someone the whole time and we generally always wait for each other at a few pre-agreed / known spots to keep the separation from getting too big.
No one is super killer, but we do push each other -- that's kind of the point I suppose.
On a more personal note -- no one has had to wait for me in the last couple months!!!!