stuff like this doesn't bother me. I'd much rather they just ride ahead of me rather than jumping on my wheel as I pass and laboring along behind me.
sometimes the other rider will be going at a little slower pace than I'd actually go but I stay back rather than passing just so they feel better. Eventually I wait until it's a nice open stretch then I'll pass with a hello or even take a moment to chat if they seem so inclined (and have the lung power for it).