I ride a lot of long days, 60 to 100 miles, sometimes pretty remote and in harsh weather (Rocky Mts). My saddle bag is just large enough for tube, patch kit, and multitool.
I've always needed a rear rack for most of the cycling I do--commuting, shopping, touring. So I just use one lightweight pannier (Arkel Drylite) on day rides. I often ride in group of up to four, so I happily carry sandwiches and extra jackets. One person in the group uses a hydration backpack with extra room for lunch and jacket.
A good friend got tired of wearing his small backpack and he went the route of seatpost rack and rack top bag.