I'll just chime in here with some of my experience this past years. I started riding again in May of 2014 after 20 year biking hiatus. My first bike out of the shoot was an s-works stump jumper. I thought I needed a mountain bike and could swap tires to ride on the road with my husband. He rode a Trek FX 7.4. After a couple of months of riding and not being able to keep up with him, I went and bought a Giant Escape 1 and upgraded it. Going to the Giant was a revelation it made the MTB feel like I was driving a jeep versus a sports car.
So, I went on riding the Giant for a while but got the itch for a new bike. I proceeded to do custom bike. Enter my Lynskey Viale. Road bike frame built with a "flat bar". Wow, what a difference. I was sooo much faster on my new bike. Not sure why though. On my Lynskey I'm 2 - 3 mph faster on average than on my Giant. Now my partner couldn't keep up with me. I just did a charity ride a couple months ago and almost kept up with the roadies.
On a whim I bought a used Trek Madone. OMG, what a revelation. I thought my Lynskey was quick, the Madone is FAST. It gets up to speed soooo quickly and I find it so much easier to maintain a higher speed. On sections of road where I was averaging 16 - 17 MPH on the Lynskey, I'm doing 20 on the Madone and it seems easier. So we bought my husband a 2016 Cannondale Synapse Carbon 105. Now we both ride fast.
Anyway, my point in this is that maybe it's not just you. It could be the bike. I'd posit that perhaps the reason why road bikers are much faster and can maintain higher speeds is probably due to riding position. Yes, you're hunched over the front of the bike but I think it makes you more aero. Also, I think with that type of riding position, your pedaling is different. Like you push your muscles in a more efficient manner. I'm just guessing here. I know that when I ride my road bike, different parts of my lower body feel like they are getting worked and again, it just feels so much easier to maintain a higher speed.
Just my .02....as long as you're riding a bike, do it so that you enjoy it. If you hate it, you'll not want to do it. So if you want to do group rides, find another group that is slower. I know that with some of my friends, they ride at a snails pace (think 12 MPH), and that's ok. I enjoy being out with my friends and just pedaling and enjoying the scenery.