That is, in no way, shape, or form a worry. I've ridden rim brakes down to -40C, so unless you're in the arctic circle, you're probably not going to see worse. The worry with rim brakes is they get clogged with snow. If that's the case you're generally not traveling quickly anyway (riding through snow is... hard) so it's not a big deal.

My disc brakes have never frozen either. Some hydraulic fluids freeze around -56C, but I've never been in that cold of weather, and neither should you! It's dangerous out when it's that cold. You certainly shouldn't be riding your bike at that temperature.