This topic always comes up.
I *never* let strangers draft me, and I never draft people I don't know. If someone tries it I shout "don't ride up my a$$", and if they carry on I pull over and let them past.
It has nothing to do with free pulls or anything like that. It is purely a safety issue. There is almost always traffic around, I ride pretty fast, and I do not want some complete stranger sat 6" off my wheel. Maybe they're very proficient and maybe they're not - in that situation I do not particularly care to find out. It only takes one wheel touch to send us both flying, possibly under the wheels of a truck or at least into the deck at over 20mph.
Similarly, if coming up behind someone, I either hang 10 feet back, or overtake smoothly - one or the other.
This also has nothing to do with being an unfriendly, arrogant roadie. I am a very friendly cyclist, but sneaking up annanounced on someones wheel and causing a safety hazard is not a great way to start out.