I'm at a loss as to what I should have done differently.

I had slowed way down in order to pass safely. Families don't really like people blasting by their inexperience children.

No passing at all while on a MUP? With the turns and hills, I would not be able to safely pass a group that size and speed, while accounting for riders coming the other way at 20mph. Not sure that's how MUPs are intended to be used.