Specialized makes a bike that is more upright, has carbon fiber fork, seat stay & you can still put a rack on the back for panniers.
I have a Roubaix & have been very happy with it. Like someone else said if you go with a 28mm tire that will help on the gravel.
Today we road 3 miles on hard packed gravel & I didn't have any trouble. Sure it won't be a soft as a ride but I made it.
Last year I rode my hybrid back & forth to work a few times & my road bike is so much more comfortable. I don't have to put near the effort to make it go like I did the hybrid. I do have to say the shock does make it a lot softer ride but I still like the road bike.
Good luck in whatever you choice, but fit is the most important part period.