I have lots of cycling, skiing, snowboarding and hiking/backpacking stuff, so if it's not icy I can find stuff to wear that will keep me reasonably comfortable on a ride. I think my record is around 8°F, though that required "extreme" measures (e.g. insulating wind-front tights under hiking rain pants, chemical hand warmers between my cycling bootie and the top of the toes of my shoes, etc.). Normally the greatest effect cold has on my rides is the length (time) of the ride. Riding in the cold + riding with multiple layers is just harder and slower, so 2 - 2.5 hours is about as long as I want to be out once the temp. is below 15°F or so (mid-ride).
I find riding at low temps much more draining than skiing, for some reason. I'll ski all day with no problem with the temps well below zero, but a couple of hours on the bike in temps in the teens is enough for me.