I'm not sure what you're expecting - if the person in front never falls back behind me while I'm drafting them then we don't switch. That's how it works - the person in front has to fall back behind and draft behind the other person, or they never switch positions. Perhaps you're not doing that.

I tried saying something to some I'm drafting behind before, but it's akward, partially because of the positioning (sound travel-wise, it's hard to talk to someone in front of you because they generally cannot hear you unless you're really, really loud, then it seems like you're yelling at them). And because the spandex wearing "because 20mph average is NOT fast" types that go fast enough for me to draft behind seem to prefer not to be bothered with chit chat - they don't really care if I draft them as long as I'm not to close, but are irritated if I'm breaking their rhythm to "chit chat". Not all of them are like that, but enough that I'd rather not annoy them even more.

Sounds to me like you just don't know the etiquette to use to switch off who's drafting who. :-P Also, the fact is it's just difficult to talk to someone who going at high speed on a bike...