I admittedly do not like group rides for a few reasons. However, in your case, why not make the slower ride more difficult by riding a heavier and non-aero bike like a mountain bike? You will surely reduce your average speed by 3-4 mph and not have to stress about it. After all its all about watts with your workout.
Another thing to consider is telling the group beforehand that you will hammer up the hills and double back to rejoin them. I have used both of these strategies and they work without issues or feeling like you cheated yourself.