Maybe there's more than meets the eye here.
If this were simply a cookie/charity ride, then you done fine. I for one always ask for permission to join what looks like an organized paceline. I've not always been given permission, which is fine as I would have slowed them down.
If this were a timed competitive event, then perhaps you were wrong. The rider was trying to improve his own time. Does Cavendish ask for permission to join a sprintline being formed by an opposing team? Isn't it the group's responsibility just to deal with it?
It kind of smacks of the Cinzano treatment if this were a competitive event.
__________________
Momento mori, amor fati.