It bugs the heck out of me when people latch on to me after I pass, without saying a word. I've started asking people to please stop drafting me, I dont want to worry about them rear ending me, not to mention enjoy biking solo. It lets me concentrate on my own workout and I can enjoy just being outside away from people.
I also think drafting is a bit silly in this context (complete strangers, not in a race, and you're not conversing with them at all). It seems like you're simply cheating yourself out of doing all the work to get from point A to point B.