If falling off the bike is the issue to hit your noggin, why in the world don't you wear one all the time? Not sure, but my head seems to be a lot further from the ground when I walk. If you take it to the logical extreme...
By the way, the only time I have wiped out serious was in suburbia - got hit from behind from a car.
Of course, down this way, we have sidewalks, roads, etc. that disappear over night - you always assume the worst with your fingers on the brakes. You keep your speed down too if you have any sense.
I feel much better without the helmet - the helmet irritates me, distracts me and perhaps might encourage me to do things I shouldn't. However, this is not saying it might not be a decent idea to wear one.
However, I am also aware that people think you need hand rails on all stairs, sidewalks with no cracks, etc, etc, etc, none which seems to be critical to survival.
At some point you just assume life is fatal. However, take my statements with a grain of salt, realizing I moved from the first world country to a third... I might be a little more risk tolerant than most. ;-)