I don't mind people drafting off me; I appreciate them letting me know they are there. Never hurts to be polite.

Usually if somebody has caught up to me, they are faster. They may just want to catch their breath by drafting off me for a bit. I once had a wheel sucker that snuck up on me and sat there for a couple of miles. I moved over, slowed down and struck up a conversation. Turns out he was a newbie and kinda lost. We rode together for several miles and I got him steered in the right direction. I also gave him a bit of friendly advice about drafting off strangers which he appreciated.