Here in Kansas it gets pretty cold in the winter and being a old fart I dislike being uncomfortable. Usually wait till temp. get above 40 degrees. Then I just grab something out of the closet, makes no difference to me if it's cotton, poly, or 100% wool. Put a wind breaker over it and I'm good to go.
Summer means hot temps. so I almost always wear cotton. To me nothing feels better (well a few things)than breaking a good sweat and then feeling the wind going thru a wet cotton T-shirt. (cools me down good)
I may be a hick....but I'm a comfortable hick!