You basically have about 4 conditions:

warm day - 1 layer
cool dry, non-windy day - base layer and jersey
cool wet and/or windy day - base layer and jacket
cold day - base layer, jersey and jacket