Sounds like you were acting unpredictably - speeding up, slowing down, etc. From his POV, he may have thought you were purposefully screw-ing with him.

I'd say you were both at fault. That is a possibility, afterall. (So is "nobody" is at fault).