It's a piece of outdoor equipment. How's it going to hurt it to be outdoors?
The thinking behind the don't-let-it-warm up theory probably goes back to when most bikes were steel. Moisture would condense on the insides of the tubes, where it could cause rust or run down into the bottom bracket. I don't know that it was ever a problem real problem.
Certainly, cold isn't going to do any damage. My single speed is a 20-year-old Trek touring bike made of Reynolds 531, and as far as I can remember, it's never been indoors. Before I converted it to SS it was my main bike for years, ridden regularly in temps from 100+ to the 20s, and stored in an unheated shed year round. I greased the BB once, at least 10 years ago, and haven't given it a thought since.
As an aside, the bike doesn't know if it's warm or cold. Leave it where it won't get stolen.