As noted, a lightweight cycling jacket should roll up and fit in your jersey pocket with no problem.

I usually put the jacket in the middle pocket and load up the side pockets with snacks. For really long rides, I plan stops to purchase additional food and water.

Top tube and handlebar bags are also quite popular among gravel riders.