Group riding is a learned skill. You can ride with faster people by being smart. When the other riders are going fast, don't take pulls. "doing your share" doesn't help if you blow and get dropped as a result. If the group ride is competitive then there's no reason at all for you to be on the front, just like in a race. If everyone is cooperating and doing a rotating pace line, you either sit in at the back and let each rider in to line in front of you, or participate in the rotation but take a short token pull and rotate through.

If the guys tend to go too hard at the beginning, go to the front and set a "false tempo"- fast enough that they will be content to sit in but slow enough, and steady, so you can warm up.

Agree ahead of time on places to regroup. Usually at the tops of climbs.

Most riders half-wheel (keeping a little ahead when riding side by side) because they are not paying attention, not because they are intentionally being rude. Sometimes a polite reminder works.

These guys may just be faster than you are. You can still ride with them by letting them know ahead of time that the ride will be a long endurance paced ride. If they're not interested in that then they won't show.