I'm in the all wool all the time camp. Not sweaters, just smartwool shirts, alone or w/ a coat as conditions require. Below 20F I need 2 shirts (usually T + long-sleeve) and close the pit-zips on the coat, and I'm good to go (seldom out for more than 45 min). In the summer it's just a t-shirt
I don't get the sweater thing. Too much fabric.