Yen, you're quite right, of course, to ride the ride at the pace you need for your training purposes. And of course he's talking nonsense about your working too hard. If you are maintaining the effort level you need to maintain, it doesn't matter whether you are at the front, off the back, or somewhere in between.

Personally I wouldn't invite others to accompany me on such rides unless there was an agreement that we might end up riding at different speeds and regrouping at some designated point. Or unless the other riders were explicitly happy to ride at your pace, rather than theirs.

This can often be a difficulty for those who ride with their partners. I have a couple of friends who ride together most of the time. The woman is a much stronger rider than her husband, and because she doesn't like to leave him in her wake (or, more accurately, because she knows he doesn't like being left behind) she rarely gets the chance to ride at the pace she wants, and needs, if she is to improve. They are having to negotiate a routine in which she sometimes goes out without him, either alone or with some stronger riders, so she can do some more challenging rides. It's a situation one often hears about the other way around, with the male being the stronger rider. Seems more complicated when male vanity is involved...