Yup,
that and layering. As you warm, take stuff off and thrown it in the pannier. It's that or misery (or sweat like mad for over an hour), take yer pick.