If they are close to their limit they can't put out enough power to maintain the speed and they end up slowing the group down. It just ends up being disruptive to a group who may have been rolling along at a nice rhythm.

The riders in the group may have had negative experiences riding in a paceline with strangers. It could range from not being able to hold a line or not holding a steady pace or surging or any number of irritating riding habits. You'd have the same negative reaction if you barged in uninvited to a foursome of golfers and asked to play with them.

When 3 riders are rolling along at a decent pace they are getting the exact workout they want. If 6 random strangers insert themselves into the mix it completely changes the workout. For the original riders to get the same workout (similar avg power) the pace needs to increase substantially and it becomes a different type of workout where you go much harder when you are in front but have more time to recover.

edit: if you and your friends are riding at a 'chatting' pace then it doesn't matter if others join you as you are just enjoying the ride and don't really care about pace. Not everyone riding has the same objectives for every ride however.