Cold weather isn't 50's. Heck, that's still summer here (Minnesota). The rule in our house is shorts if the temp is 50 or warmer. I'm not going to tights until around 30's, maybe low 40's if it's windy or really damp. I'll wear knee warmers down to 50 just because I have a bad knee and it complains if it gets cold.
I've ridden in the winter before, but this year I'm getting a fatbike and I'm going to go hard at it all winter. Winter here, it's not unusual at all to get -20 (i.e. normal) and we hit -30F enough that it's not a big deal; a curiosity for sure, but not a big deal.
That all said, degree for degree doesn't match around the country. A 0F day in MN is colder than a 0F day in Colorado for all intents and purposes, for example.
J.