I don't think it is worth not to wear a helmet. Skull fracture are always bad. It happens easily.
Once every three month I leave my house lock everything and realize that I left my helmet inside. If I really go round the corner I don't go back to get it (unlock the house, undo the alarm, etc). Even then I feel it is stupid not to wear it. Any other times I wear it.
Have you noticed? Over here, you seldom see a motor cyclist with a helmet (even though it is a legal requirement) but most cyclists wear one.