This ^. Well said NatUp.
We've beat this a bit to death, but we only have the OP's side of the story. Maybe he could keep up with them because they were on a slow paced ride, or felt they had to take it easy to accommodate him. Or maybe he's crazy fit and could keep up with them. And there are MAMILs (Middle Aged Men in Lycra - a funny term I hadn't heard before and hey I'm middle aged!) that are all hat and no cattle. On one of the group rides I went on that was billed as brisk (not crazy - 18 mph in the flats) an older guy showed up on a mountain bike with slicks and Fredly kit (he had a regular helmet though). Well guess what? He got dropped. Well the route passed near his house and he dropped himself. Nobody was rude and people checked in with him to make sure he had a plan, but I doubt he'll be back for that ride until he has a road bike and maybe some more training. There are rides billed as easy and moderate that he could try. The rudeness the OP experienced was inexcusable and I wouldn't want to ride with people who behaved that way even if I did fit their mold.
I have an old mountain bike I use for short commutes and utility purposes. It's got platform pedals, rack and panniers, fenders etc. I jump on it in whatever street clothes and shoes I want. But when I go for a group (or solo) ride where the purpose is to ride and ride hard for 1,2-3 hours I wear cycling kit. Sure it helps me fit in, but it is also functional. I eschew corporate logos and neon colors (not my style) but hey that's just aesthetics. Clipless pedals are awesome and I wouldn't want to ride without them.
There's a group for everyone (I hope) but not everyone is for every group.