During the winter i find that the combination of a sweater and a scarf is perfect for riding.
The sweater allows enough wind to pass through so i don't get too hot while the scarf keeps my neck warm. If your neck is warm it is amazing how warm it makes the rest of your body feel.