Like Tarwheel, I hate the heat a lot more than the cold. In cold weather, I'm uncomfortable for five minutes until I warm up. In the heat, I'm comfortable for five minutes until I warm up. Heat also means that I have to commute in cycling clothing, which I hate doing. I'd rather just get to work in my shirt and tie (and pants).
I generally go all year round, unless there is a lot of rain. Because my commute includes commuter rail, I do not like to get on the train and sit down next to someone when totally soaked.