Drafting without the other person knowing seems like a pretty dangerous thing to do. When I am just out riding, I am not exactly concentrating on making a perfectly straight path. I might zig or zag as I wipe sweat from my eyes, grab a drink, avoid some dog poop, or just for the heck of it. I am not sure if I have my physics right but it is probably more dangerous for the guy in back but I would still be ticked off if we crossed tires.
I do have a mirror, though, so not too many people are going to be able to sneak up on me.