I think you have been unfortunate with your group riding. Admittedly you may want to go slower than the others, but there is no excuse, for them to make you feel uncomfortable.. In the group that I ride with, admittedly off road and mainly solo's, we have one rule. First one to the top of the hill gets cold waiting for the rest.
We do have newcomers ride with us and we do not know their standard or their stamina. Hills are where it takes it out of you, and we ensure that at least one rider will stay with them, no matter what their standard. Generrally, this is to slow them down so that they can do the 30 miles that our normal ride lasts. If they are "Ultra- fit" then we generally have a couple of riders that can chase them to stop them from getting lost.
My suggestion is that you find a different group to ride with. If you can't find one, then start your own amongst your friends. It is a great way to get your friends intersted in doing some form of physical activity that will improve their health, aswell as get your fitness up so you can poke two fingers at the other group when you see them out on a ride in the future.