My compromise is a combination of a larger road seat pack (say 1.5 - 2.5 liters) along with a handlebar bag (I use this one from Jaand). The handlebar bag works to hold extra supplies without interfering with lighting/electronics, and can be easily removed for rides where it's not needed.

Like most people here, I use lightweight cycling-specific wind and rain jackets that can be stuffed in a jersey pocket. I also ride when it's below freezing, and the handlebar bag is good for stuffing items that are not cycling-specific, like a winter softshell, extra baselayers, extra gloves, and chemical warmers.

This is more of a gravel than a road setup, but here's the idea: