My saddlebag is way too full for even a thin jacket. I carry spare tube, patch kit, 3-oz bottle chamois lube, wet wipes, spare blinkie or battery, emergency clif bar/shot blox, CO2, levers, wallet/ID/insurance, keys.
My jersey pockets are busy with 5-oz flask of sports gel, cellphone (running cyclemeter), clear safety glasses (when ride continues after sunset), possibly a park gate pass or work badge to get through gate. Only things that I want fast access to on the roll.
The windbreaker stuffs into its own pocket and has an elastic waist belt with a metal buckle. I usually use a velcro tie on the buckles to fasten it to the bike saddle over the saddlebag.
Alternately, I'll use a thin drawstring backpack. If I'm bringing along a windbreaker, temperatures will be cool enough that having an ultralight pack on my back won't get too hot and I'll probably have tights or other apparel I'll want to ditch or swap out for lighter versions.