I think you misjudged what he said. After reading your post twice it seems to me he was just trying to tell you that you could go faster if you wanted to:

1) He doesn't know you're tired
2) He thinks you may have been holding back to not drop his ass so was doing you a courtesy.

I can see how his statement could've been miscontrued as something akin to a taunt. IMHO it wasn't.