I've been drafted several times this summer on my commute home. We get a pretty bad west wind this time of year almost every afternoon. There's been a number of times when somebody called out to pass me and they're right on my back wheel and I never noticed it, or I just happen to look to the side and see a drafter in my peripheral vision.
So far it has not caused me any problems, so I honestly don't mind it. It seems like the riders who have done that to me are experienced at it and know what they're doing - which might be a horribly bad assumption on my part.
One time I came to a stop at an intercesection and a guy was right there with me and said, "thanks for the pull..." again I didn't know he was drafting. I said, "hey, no problem, I'd dig in and go faster if I knew you were there". He said, "that's alright, I appreciate commuters...... on windy days..." I didn't know how to take that honeslty, but I thought it was kinda funny.
btw - I would never try to draft a couple riding together for the same reason I would never plop my lawn chair down an inch from their picnic blanket.