If I may, Mithrandir: group rides (and races for that matter) have a certain ebb & flow to them.
Usually, after warmup, they start off fast but then settle into an easier pace. It sounds like if you could have just made it through that hard part at the beginning, you might have stayed with them for the whole ride.
Keep it up. Keep going back and trying hard.