This is difficult to explain, I guess.
It really wasn't an attack, it was razzing, by which I mean it was joking with a point to be made. Sort of a pushy joking, I guess. The subtleties of language are almost impossible to corrrectly express on the internet!
Anyway, thanks for all the responses. I will take one suggestion - I will see this guy tomorrow, and will ask him if he would like to go for a short, fun ride with me. I will be curious as to what he says!