If all your thinking about is crashing, you probably shouldn't be racing. One, because it won't be any fun, and Two, hesitation from fear is dangerous.
That said, don't delude yourself about the relative risks of training rides and other events. Some of the nastiest crashes I've been around have been on trianing rides, and organized centuries.
While the risk of donating skin, a broken collarbone, and/or some cracked ribs is higher racing, than doing training rides, and centuries, I'd definitely be willing to bet that the risk of death is higher riding on open roads, than it is racing on a closed course.
__________________
You could fall off a cliff and die.
You could get lost and die.
You could hit a tree and die.
OR YOU COULD STAY HOME AND FALL OFF THE COUCH AND DIE.