I've found that a windbreaker is adequate over a jersey and arm warmers unless it really gets cold, as in below freezing. Then it's a heavy rag wool sweater over the jersey/aw combo with windbreaker on top.

I've become a weenie lately and seldom ride when it gets that cold but it has worked well in past years.