Cooler temperatures will increases the amount of heat lost (if you are not clothed well) but has a very minimal effect on caloric burn rate; as most of the heat would have been produced anyways. Your body just reduces sweat production; and thus the amount of thermal losses due to evaporation.

However, shivering increases the rate calories are burned to 500 - 600% above normal.