I agree plus he did pass you first then you passed him.
Originally posted by jmlee
Maybe you should have pulled out to the left to signal that your turn at the front was over. This is the normal routine for a paceline--the front rider determines when her/his turn is over. He may well have recognized the signal. You could have supplemented it with a friendly, "your turn." My guess is he would have obliged.
In any case, I don't think should be too hard on the guy without having made a clear signal or directly asked. Heck, he might have been wondering the whole time who the superman was in front of him who liked to take such long pulls.