I wear mine most of the time cause there's not really any good reason not to. A helmet with good ventilation isn't all that hot and they're fairly comfortable. I guess the only negatives are helmet hair and the rank smell if you don't wash the little pads.

EDIT: and they look dorky if you care about that kind of thing.
DOUBLE EDIT: Yes, I know I'm not wearing a helmet in my avatar. I said most of the time.