I understand very well your problem. I feel the same way sometimes. Generally I'm slower than most men and I really don't mind when someone passes me. But sometimes I feel they're angry at me for "being in their way" and just tailgate me way too close until there're no cars on the road and they can pass while giving me an offended, angry look. Other times I've passed a cyclist on horrozontal terrain but they've caught up with me up the next hill which ultimately has resulted in the same thing.

I haven't said anything so far. IMO nothing rude would have an effect. Maybe "you were following me way too close" or "don't ride so close near me"