I really don't worry about people that do that crap. I compete against myself and the clock, that's all.
A friend of mine plays that game. Anytime someone passes us, he drops a few gears up and pours it on after them, as if he's making a point. I keep pulling along and pass him in a few moments as he's gasping after his sprint.