This used to annoy me and then I got over it.
Nowadays I will either look back, make eye contact, and keep pulling the other guy. If I am not in a mood to be around other folks I will sit up, slow down, and maybe wave at the other rider to go ahead. I have better things to get stressed over.
As for me, I prefer to pace other riders I do not know instead of drafting. I would rather draft off riders I know and trust.