The answer is always some kind of layering, and storage area for what is not in use. Clothing with wind block at the front does help. My upper outer is almost always a thin wind breaker type jacket. Most of the time, it is an inside layer that is removed when needed. I live in south coastal Delaware, 5 miles from the beach, temperature and wind fluctuates greatly from beach area to inland area, and a huge difference in riding with and against the wind. If one gets sweaty enough to be damp, one can get quite chilled.