I wouldn't have done anything. It sounds like you're overreacting. If it bothered you that much you could have stopped and taken a 5 minute break so you wouldn't see him again.