I've never had it happen during a normal ride but during charity ride or group ride, I dont mind per se but I appreciate it when/if they take a turn too. The very few times that I've done it, I make sure to reciprocate as well. I never felt a difference when there were guys in my draft.