IME, the hardest part about cold-weather riding is overheating. It's easy to stay warm once I get going, but if I start to overheat it's really uncomfortable. Even on sub-zero nightrides in winter, I rarely use a full jacket for long. Vests are a good balance for me, especially layered with thermal jerseys.