I second, or third, getting an Endurance framed bike. I have one which I use 90% on road, 9% on gravel and 1% on sidewalk.

I run 28s but it can easily handle 32s if I were to do more gravel. The frame is more relaxed/compliant for a smoother and less high strung ride than a pure racing road bike, which I feel you would appreciate on longer rides. I also have pure road bikes and a mountain bike for comparison and feel the endurance or a nice gravel bike would fit what you are looking for. (Would definitely veto a hybrid).