Unethical? Yes. Why should I do your work for you? You latch onto my wheel without notice and assume that I'm your servant?

Unsafe? People will tell you that the guy in back is the only one who can go down. That's not at all true. Get your front wheel into my rear derailleur, and it'll be a mess.

Were you wrong? Yes. All you have to do is say something. "Mind if I sit in?"
My stock answer is "I don't want you there. Thanks for asking."

If I'm riding by myself it's probably because I want to ride by myself. I don't want to have to think about YOU when I change course/speed.

It's funny that people assume that because it's more efficient to draft each other, everyone wants to. Cycling is not an automatic village on wheels.