I guess my fear is crashing bad a la some of the photos I've seen on this board. I don't worry much about a cager mowing me down, if it happens, it happens. I've been living with the knowledge kidney disease will probably take me down, maybe at age 40 (I'm 35) so I kinda feel as though time is running out, anyway. I don't take stupid risks though.
I more fear ghastly crashes with ugly road rash/broken bones/nastiness. I've crashed twice and nothing seriously bad happened. This made me less afraid, crashing doesn't always mean you're going to be FUBAR.