In addition to "how cold?" you also need to specify "how far?" I've got a graduated system of clothing for my current 10 mile commute. When my commute was only 5 miles each way, I usually managed to get by with clothing about 10 degrees warmer than what I wear now.

Significant thermal mass also plays into things, along with metabolism. Some people are so thin, and their metabolism is so low, that they're wearing full skin covering while I'm riding around in shorts and short sleeves.