When I'm riding with a group on a road ride, we draft. That means staying about 6" from the wheel in front. Any farther and you're not getting much benefit. It also means that everyone in the group understands that we're drafting, knows not to make sudden moves, knows that the others are similarly skilled in bike handling, etc. I do not draft people I encounter on my commute because I don't know they're bike skills, they don't know mine. I think it's kind of rude and unsafe to do in that circumstance. To then draft without the lead rider's knowledge is idiotic. Drafting adds risk to the ride, so sneak-drafting puts the other rider at risk when he's not even aware. What happens when the drafter closes the gap a little too much, overlaps the front tire with the draftee's rear? In a group ride, you can expect that the lead rider is trying not to make sudden moves. The sneak-drafter has no idea when the lead rider might swerve to avoid hitting that spider on the path. It's just stupid, the OP has a right to be pissed.

The one sure way to attract sneak drafters though, is to pass roadies on your commuter like they're standing still. They will haul ass to catch your rear wheel and stay there as long as they can. Makes the average speed for their ride look better. Had a couple of those last week, scared the crap out of me when I looked back half a mile after passing them and saw them right on my wheel. So I dropped them.