I say keep riding with them. You'll improve much faster if you are pushing yourself, which will happen when you ride with a faster group. Let them know from the beginning that you'll be ok on your own, and to ride at their normal pace if they should drop you.

Screw what they think about your bike. If anything, that should give you more motivation to be a better rider. 99% of my road biking has been done on my simple aluminum Trek 1000, while I ride with guys that have $5000 carbon fiber racing machines. I find great joy in cruising past some of those guys who let their wallet do the talking more than their legs.

At the same time, start putting away a bit of money for a faster road bike. Riding on your heavier bike will make you a stronger rider once you move on to a lighter, faster bike.