Chest hair? You might try a wool jersey with arm warmers for those brisker days. Wool will keep you warmer without the extra layers. If you're just a bit cold at the start of a ride, that may well be OK because after a few miles your core temp goes up and you'll feel fine. A nylon wind breaker very foldable for that jersey pocket might also be a way to start the day.

Otherwise, some sort of thin, long sleeve tech turtle neck with zipper is useful.