Good on you for not wanting to be an a... There's really not much you can do if others want to be, but there's no reason to expect so much animus.
If I pass someone who doesn't seem to be in a friendly mood, but they "take it as a challenge" and hop on, I figure they are engaging and friendly after all.
If you end up passing the guy you drafted behind for a while, I don't think you can go wrong by saying "thank you" or "nice pull." Normally I'll try not to go too much faster at first and give such riders the opportunity to take a draft from me, but if they aren't in a friendly mood, they drop themselves. Nothing awkward about that.