It only happened to me a few times. I usually get my workout on the way into work(4am), and on the way home, it's rush hour and more of a leisurely commute. But there's been a few times when I want to get a workout in the afternoon and takeoff for a rural road. Once a roadie asked before drafting. Instead of being annoyed, I tried to take it as a compliment. The few times when one would just magically appear on my 6, I would maintain my speed, but just swerve ever so slightly so they'd get the hint. Especially when I'm doing intervals, I don't need some creepy biker that close to me with no mention.

BTW, the one time it was a unannounced female drafter, it wasn't so creepy.