Lucky me, the only flying things I have to deal with are bats, falcons, herons and the occasional peacock. Only a falcon has ever stayed with me for long, and when he decided to take a pull and I fell off the back when he increased the speed. (Falcons don't offer much of a draft.)
If someone was harassing me with a drone and it was creating a hazard, I might make the effort to find the operator. I'm sure that after a pleasant conversation s/he would see that endangering me was probably not as fun as it seemed. I wouldn't threaten harm in any way. I think people sometimes forget that pulling the public into their games can be misinterpreted and I would try to help the operator to have a bit of empathy. I get it that these are fascinating, irresistible toys and good people will occasionally make mistakes in judgement.