In an organised ride situation, riders are set off in a group, and pacelines will likely form, there will likely be drafting, and there could be some pushing and shoving from the sides. I'm not fond of being set off in a large group for that reason.
However, once you get out on the course riding by yourself or a friend or two, drafting a stranger unannounced is frowned on. And it is especially bad if you draft a stranger for a while, and then suddenly pull around the stranger and disappear in the distance. How rude!! And that has happened to me twice!!