I purposely load my bike down to the point I probably ride at a pace similar to yours(as I don't have a computer I never really know or care for sure) .I ride for fatloss ,adventure and training for touring.Speed is not really a primary objective though my pace has improved significantly with training and if I ever got a light racing bike I am sure I would be much faster.I don't usu ride with a group but on the routes I take there are usu a lot of group riders particularly on weekends and there is still a lot of comraderie even though I sometimes take other routes on weekends as they can get overcrowded with joggers lol. So ride at your own pace and for your own objectives and if you can do this while maintaining the comraderie of participating in a group all the better but don't let trivial things like that determine whether you decide to cycle or not as it doesn't sound like you are.