If you have any general concern toward the safety of other humans, it puts a responsibility on you.

Frankly a lot of people who hitch a wheel without asking have flat-out bad judgement. Mostly because they actually trust a stranger who hasn't made any kind of demonstration of safe or fun group-riding behavior, but for other reasons as well.
Recently I had some guy on an ebike do it to me. He was going around 15mph, I was cruising along at around 21mph, and he hopped on my wheel when I passed. About a mile later, he decided to hop in front to "help" right as we started to go downhill; I had to immediately re-pass to get up to reasonable speed. Not a big deal, but kind of silly.