That balance is hard. In order not to be cold while riding, you need to wear fewer clothes than you would while walking. But carrying the extra clothes on the bike is an unpleasant chore.

I wear as many sweaters as I need, depending on temperature, and I wear a windproof rain jacket on top.