You need to look at what you're doing to get ready. I can be out the door in 10 minutes, and I have an 11 mile ride.
I ride all year round in Michigan, so I say "never". The coldest temps so far have been -25*F. The only day I think I probably should have driven was when the snow was so deep that I had to walk up every little hill, and I had to pedal pretty hard to get DOWN hills. Luckily that was only for a couple of miles then I got to plowed roads.

I don't bother carrying water; my ride is only 40 minutes or so. If the temps are over about 95*F I will carry water. I used to carry it but a few winters ago I just got out of the habit since even if I put hot water in the bottle it'd be frozen solid halfway through my ride.