I tend to take my helmet off for the last mile of my commute in the mornings, during my cool-down period. I like my hair to dry out a little bit in the last mile. That area, for me, is a VERY sedate backstreet through the business park with almost no cars at all when I'm coming in. So I don't feel unsafe [or I wouldn't do it]. I actually strap the thing to my rack, just to keep my hands extra free, just in case.
However - I forgot my helmet yesterday [no freakin idea how that happened, I just walked out the door without it and didn't realize for 4 miles]. And um... I'm alive today. So that's nice.