He was probably trying to beat a Strava KOM time by 2 seconds, and your simply being on the trail cost him the title.
I wouldn't have played his game of hopscotch.
I can usually lay down a bit of heat, at least for short distances. If I really wanted out of the situation, I'd try to get far enough ahead to take a corner out of sight, and let him try to catch someone who isn't there.
Or, when he is leading, let him take a corner, then quickly turn around and go the other direction for a couple of miles before resuming my ride in peace.