Typically the only trips I make sans helmet are like half a mile away. It's only got a few intersections in that space and traffic generally crawls along anyhow.
And definitely on the mountain biking versus road thing. Dirt riding was why I started wearing my helmet again in the first place. Then I decided if I could wear it out there, I could be a big boy and wear it around town too. The only time I've ever hit my head in a spill was on my mountain bike. Bombing a rocky trail full of babyheads and one of 'em slip. I went over the bars and did a face skid. Luckily my visor also ripped loose and came down, so rode that along and only got a little scuffing on my lip and chin.
This trail right here, in fact (the part I wiped out on is a few feet in the opposite direction and not quite so gnarly, ironically enough):