If I make a pit stop at a 24 hour big box, I almost never take it off because ... if I'm wearing it, no one can steal it (unlikely but hey, anything can happen in this world) and more importantly, I'm often a little "foggy" for various medical reasons, and I tend to set things down and forget them -- yes, I have had to ride back 3-4-5 miles to retrieve a helmet (and a wallet, and keys, and a phone ...). So, one less thing to worry about losing that way.
I also agree with the DrIsotope - if you're uncomfortable and want to remove your helmet at the first possible second, it's probably not a great helmet.