I got a down vest at Costco for $35. It stuffs into an old silnylon sack I have, and that fits securely into a bottle cage. Below freezing, sometimes I'll wear the vest over my jacket.

When your core is cold ("upper body a touch chilly but acceptable"), your body wants to conserve heat for what's most important and that means less for your extremities.