Rob e's comments pretty much sum up my opinion as well. Having a cold unpleasant nights sleep certainly isn't worth it.
Also don't forget the very real possibility of riding all day in rain and wind for already feeling perhaps cold and miserable.
+1 on having some warm socks for a cold night.