I've gone for a ride in single digit F temps many times. For me, it's not so much the temperature that limits me, but the conditions of the roads. If it's precipitating, and the road surface is questionable for a (road) bike due to ice then I won't go. But if it's dry and no chance of ice I will go. Obviously, even with extra layering, the temp sort of dictates how long I can stay out. When it gets down into the low 20s and teens I'm only good for about 90 minutes.
I've also learned, the hard way, that when I do ride in those colder temps that I have to be careful where I go. I once got a flat out in the countryside when it was about 20F. I took my gloves off to make repairs and my hands got too cold for that very quickly. And it's impossible to make those repairs with gloves on. When I tried to call my wife for a pickup I discovered that I was in a cell dead zone. I had to walk about a mile to get a signal...and by now freezing my a$$ off because I wasn't generating body heat. Moral of the story...stay in civilized areas when riding in extreme cold.

Dan