For rides that start in the low 60s and end up in the upper 70s, I have arm warmers and a windbreaker vest that can fit in my jersey pocket. If it's a colder start, I trade the windbreaker vest for an insulated vest (Q36.5 Adventure, I think), maybe throw an extra windbreaker jacket over the top. At this point, if I'm confident I'll be shedding layers, I'll throw a top-tube bag onto my bike, as well. The jacket and arm warmers can fit in the jersey, but the insulated vest takes up enough space to warrant the extra pack. Leg warmers are a bit dicier (can take up a lot of space), might be worth trading those for knee warmers and longer socks.

Also, dunno about where you are, but you may want to throw in a small bottle of sunblock for once the layers start coming off.