Arm warmers at 50° is about where I'm at, too.

Today was 32° and I had a base layer, LS jersey, vest, light gloves, light hat (covering the ears), thermal tights over bibs, lightweight merino socks and neoprene overshoes (my feet do tend to get cold). Once I got going, I probably could have taken the vest off, but instead I eased off the pace to keep from sweating too much. If it's in the 20°s, I'll replace the vest with a jacket and go to some heavier gloves (edit: the other Minnesotan has reminded me about the neck gaiter - that'll go on in the 20°s, too). I tend not to ride much below 20°, but if I do, same gear but shorter duration (90-ish minutes).