Its more than I wear on 1-2 hour rides at those temps.

Everyone has there own comfort level. You do have to be cautious when it is cold in case there is some sort of mechanical failure or the temps drop fast you do not want to get hypothermia.