I hate the guys who pull over and/or slam on their brakes as soon as you get within 20 feet, like I'm trying to draft off someone doing 14 mph. I snapped at a guy a couple of weeks ago, who saw me approach as I was waiting for a safe pass (narrow pedestrian bridge with oncoming jogger). He, immediately, slammed on his brakes and swerved in front of me just as I got within 10 or 15 feet. I told him what an @$$hole he was and that I didn't intend to draft off a slower rider. He blurted out something like "I don't like people on my @$$". I told him that if he was scared of other cyclists then he shouldn't be on the trail. I just let it go after that.
That said, if I ever crashed into another rider because of my drafting, I'd likely feel apologetic and responsible, I certainly wouldn't leave the scene w/out first checking to see if everything was alright.