dizon510 --
I am often in a similar situation riding with my coworker friends and I'm torn on the issue too. When I ride with them, they keep a faster pace and can climb better than I can. I am always way off the back. Luckilly that means I don't ever have to pull, but I can't even keep their wheel to draft. By way off the back I mean I am at least 50 feet behind them for 90% of our rides. It does drive me to push a little harder but I like to push myself at my own rates, not at theirs.
So I end up riding by myself more often than not. The way I see it, if I can't even keep their wheel then why even ride with them? There's really no point if I'm 50 feet behind because I can't even talk to them or anything.
In the end you have to do what's best for you. I agree with others who have said if you're training and/or more than a casual cyclist, then stick with them and you'll improve. If not, I'd suggest training on your own until you can perform better. My personal hope for me is that I can one day keep my coworkers' wheel and even take turns out front. It will come soon-- I used to always be 100-200 feet behind them, so 50 feet now is a good improvment!