Well now. Being passed by another rider does tend to spur me on, purely the aspiration to ride faster I suppose. Sometimes the rider is much stronger, there's no chance of matching thsier pace, especially if its a commute ride, and I have "luggage". Often, its possibe to hold the passing rider, but I'll keep back sufficiently, and slightly to the outside. This makes it easy for them to know where I am, and that I'm not drafting. Then, there is the rider who passes in a flurry of excitement, but sustained effort on my part reels them in for a re-pass, never to be seen again. Very occasionally one of these riders will draft, which is frustrating & un-nerving. A while ago, I collected two such paceline drones. I'm sure they felt good drafting a back-pack sporting commuter on a road/cross Frankenbike

To the OP - I doubt that you were irking the other rider at all - he was just staying aware of his surroundings.

Cheers,

Ed