Managing hot/cold can be tough with changing efforts on the bike.
Snowboard helmets can do wonders to keeping your head warm, but they will also cook the head on a hard hill climb unless it is very cold out.
I use a rain slicker (designed for cycling) over a fleece sweater, and ski gloves in the winter. If hot, unzip the slicker and open all the vents. If cold, zip it up. And, I find myself frequently reaching down to adjust the zippers a little. That is just normal.
Keep in mind that some body temperature variation is very normal, so one's temperature could easily hit 100° in the summer, and 96° in the winter. My hands are often much cooler than that.
However, you shouldn't get so cold as to experience significant physical symptoms.