How about a lightweight down jacket carried in a stuff sack for when you do have to stop? Lot's of warmth, little weight. Or is that not a good idea?
__________________
FB4K - Every October we wrench on donated bikes. Every December, a few thousand kids get bikes for Christmas. For many, it is their first bike, ever. Every bike, new and used, was donated, built, cleaned and repaired. Check us out on FaceBook: FB4K.
Disclaimer: 99% of what I know about cycling I learned on BF. That would make, ummm, 1% experience. And a lot of posts.