If you're riding 35 on the weekend you can do 50 no problem. IMO.
I hear you about the cold weather - it hit 57 yesterday and RAINED! I stayed in because I just finished cleaning my bike from stem to stern and I was in no mood to do it all over again. I actually like that temperature... a wind vest or a long sleeve base layer and I'm comfortable.