For me, 4 has been sufficient. For most of the winter I ride with my regular work clothes (undershirt and shirt) and a medium weight ski jacket. When it gets colder, i.e., 0 F or below, I add a fleece under the jacket. This combination has worked for down to about -10 F, and it hasn't gotten colder than that around here for a few years.

I don't happen to sweat a lot. When it's hovering around freezing, I wear a fleece with a light waterproof shell over it.