I've often thought of creating some sort of windbreak around my brakes/shifters. Like a 1.5L bottle cut vertically in half, and then zip tied to form two concave "mitt shelters".

It's often not the temperature but the wind chill that gets my hands all cranky.