Did I miss the part where the OP peeled off and gave the other guy an opportunity to pull for a while?
I mean, if I'm on someone's wheel for a while and they just keep going and going, I'm not gonna try to overtake him just for the fun of it.
Although, I might yell something about taking a turn... but maybe not(?) For all I know, he might be enjoying his sufferfest.