Most people I have spoken to say that distracted driving is the main reason to avoid cycling. Of those who do try cycling, it is the accumulated effects of distracted drivers and/or a single road-rage incident that causes them to stop.
In my own experience, I have quit bike commuting before when I've had a single, scary close call. Sometimes it just doesn't seem worth it.
(Examples: Guy pulls out of a liquor store parking lot without looking and at high speed nearly running me over. Another example: Guy crosses two-way street thinking he is crossing a one-way street. He looks in only one direction and nearly runs over me and my pet parrot riding with me.)