The coldest I have ever ridden was 2/14/2004, -4F/-20C. That was my regular morning commute, 8.5 miles. 2 days later we got down to -8F, but I was traveling out-of-state that day and drove instead.
Since I have osteoporosis I have given up riding when it is icy, so I don't get as many cold rides in as I used to. I always found that at single digit and lower temps, drivers were a little more courteous, probably they just are afraid of the crazy lady on the bike.