I have killed several helmets and usually wear one. I broke a few while mountain biking and one in a bike/car accident. So I do value my helmet. That said... I don't wear it absolutely every mile I ride. I'd say about 99.7% of the time when on tour so far.
On the TA, I took it off for an 8 mile climb in ~100 F heat on a day I was not feeling well. I don't think I would have made it with the helmet on. It was so much cooler with a cotton cycling cap on!
I went without for my long day on the SFT tour. I started out without it on an empty access road (1 car every 20 miles or so). It was so nice riding without it that I left it off all day. The experience was so pleasant that I may repeat it once in a while on future tours when the weather and the road conditions make me long to ride some helmet-less miles. It will be reserved as a once in a while treat though.
When it comes to others not wearing helmets... I don't think it is all that unreasonable of a choice and support their right to choose.
I doubt that I will ride without a helmet very often, but it sure is nice to go without once in a while.