I think you're fine with what you did. People randomly latch onto my wheel without asking me all of the time, especially if I'm riding with a buddy. I don't really mind it at all - in fact I think its great. You can politely ask them to get off of your wheel or just motion them to pull around, its not really a big deal.

Roadies (and people in general are crazy). Last week me and maybe 5 other guys were cruising down what is essentially a double-wide MUP when I overheard a conversation between 2 cyclists that we were passing. One cyclist passed the other giving him plenty of room but I overheard the other yell, "I'm tire of your **** a-hole, f-bomb you a$$hat." I asked if he knew the other guy and if they were friends joking around with one another and he said he had never seen the other guy before in his life. Most bizarre rage I've yet to see on the bike.