When I go on a bike trip I take summer clothes only...cycling shorts and jersey. Last year on a 8400 mile June 22nd-September 19th trip that's all I took. Remember your body is the best protector of itself. Let your body heat keep you warm not the clothing. The clothing will make you cold since you will raise you body temperature by wearing it. When your body temperature gets above 98.6 you start sweating to cool the body down. Sweat is called evaporative cooling not evaporative warming. It's not designed to warm you up, rather to cool you down because you are already too warm. Instead of taking clothes off and on just let your body naturally take care of itself. It generally takes 10-15 minutes of riding to get the body heat kicked in evening during the winter.
Yes, I will admit that last year on the 227 mile day that I finished around 4AM I spend most of the predawn overnight hours riding with temps in the mid to upper 50s with nothing but cycling shorts and a jersey on. I was fine as long as I kept moving. The real problem had simply been the nice warm up the previous couple of days otherwise I would have even been fine while stopped. When you get hit by 90+ degrees for daytime highs for a while when you are out riding everyday the body gets conditioned to it so when the temps drop like they did that night the body isn't used to the colder temps and it will make you colder much sooner than you normally would otherwise when your body is conditioned for it.
Just remember the key thing to do is to get riding and let your body warm up due to the exercising then you will be warm even if you don't have much in the way of clothes on. Let the body regulate itself. I do this even during the winter and only wear cycling shorts and uninsulated wind pants and a long sleeve cotton tshirts and an unisulated wind jacket clear down into the teens. I'll ride 100 mile days in the dead of winter when the temps are in the 20s dressed like this. I just let the body heat keep me warm, not the clothing. The clothing can't keep you warm. Take a dead person and toss all the clothing on him that you want and see how long he stays warm. It's your body heat that keeps you warm.
No way would I take a fleece for a summer trip unless I was traveling to the top of the Rockies or someplace like that. I have would to be way up in elevation before I would contemplate a fleece for a summer trip. Wasted pack space and pack weight.
By learning to ride smarter you can ride with less and save more money and go for longer trip because you don't have to work as much to buy the extra unnecessary gear. Ride smarter.