When I ride, even on the coldest days (below 0 degrees F), my upper body radiates heat like a furnace.
I work the hardest (and spend the most money) on keeping my feet, hands and head warm, and then the rest of my body takes care of itself...as long as I wear short sleeve shirts or a vest with arm warmers so I can get rid of the excess heat.
I wear Docker cotton shorts year around, with leg warmers and knee socks on cold days.