On a big descent it may be worth changing into a dry jersey, windproof and possibly some warm wool or fleece midlayer. You need to carry spare clothing on cold hilly rides so you probably need some small luggage container under your saddle.