I've had people hit me from behind, get their wheel tangled between my wheel and the derailleur/frame, and almost take me down. Only excellent reactions from both of us prevented a spill.

The only time I expect random people to grab my wheel unannounced is when I'm part of a large group ride, like the local club or the recent Grand Fondo. Then, it's sort of implicit. Otherwise, if I feel someone on my wheel I'll say something.

On my end, I'll usually leave a few bike lengths in front of me or pass. I haven't really drafted strangers.