Would you rather be hot or cold?
Hot of course, because nobody likes a cold hearted beatch.
In the summertime here in the Bay Area, sometimes we get a temperature swing of 40 degree. It can be 50 in the morning and 90 in the afternoon. Just last week, one day it was like 36 in the morning and upper 60's in the afternoon. So I pretty much carry what I need that can work for both extreme.
I like to have thin layer that insulate well. I wear thermal Patagonia as a base layer, then a fleece pullover and top it off with a thin wind resistant and a Performancebike Ultra rain resistant jacket. If I know it will rain hard, I use my Shower Pass jacket and that gets me very warm. Removing one layer will not have a huge effect because each of them are only doing so much in terms of warmth. In the afternoon, I just wear the jacket and a top. The other two layers are thin enough to be pack without bulkiness. Try not to have any single layer that is too thick.