Keep doing your rides with others as long as your goals are met. Possibly the concerned rider was at breaking point and needed relief from your pace. Being the ride organizer does give you the ability to establish the parameters of the ride

I frequently invite others to ride with me on various training rides. I always call out the time and effort that I'll be riding. At times the pace might dictate that I sit in to keep my HR in the zone I want and some times my riding mates will just follow my wheel. A few of my buddies Z4 is the same pace as my Z3. As long as they are drafting they can keep their HR in check. It hasn't happend, but if the others started hammering I would just let them ride away.