The coldest season I ever remember riding in was the winter of 2000-2001. I was car free, and faced a 15 mile commute to work everyday. The weather was brutal, it seemed like everyday was consistently below 10 degrees for at least a two month stretch and there was one stretch during which it was consistently below zero.

I rode in tempatures down to -14 and If the weather was calm, the only problem I experienced was the low temps seemed to cause problems with how the bike functioned, the derailers were slow, the chain creaked, and the brake cables tended to bind up.

If it was windy and cold, I hated it. I could never keep my hands, groin, or feet warm enough. Touching a freezing cold bike didn't help.

I think if it's so cold that it's adversely affecting the normal functions of the bike, than it's too cold.