You've gotta be kidding me. 50F is cool, but isn't remotely cold. That's the typical AM temperature for a ride in sunny Southern California in the spring and early summer!
You have nobody to blame but yourself if you're freezing at 50F. Put on some friggin' clothes. Until you're down in the freezing temps, it's not rocket science. And yes, you're going to sweat like a dog on the hard climbs and be chilly on the downhill. No way around it unless you choose to not hammer the climbs.