OP, I had the same needs as you mention. I wanted a bike that would do well on the road and on gravel/dirt roads. I already have a Specialized Roubiax as my go fast club bike but I would not consider riding it on gravel so I bought a Specialized Tricross Sport which is a cx bike. It's very versatile and comfortable. It can take wide tires, fenders, racks etc and does well on gravel roads. I also use it as a backup (with road tires) to the Roubiax. I was surprised that I am able keep up with my buddies in a paceline.
There are a lot of different cx bikes out there that would work well for you but I really like my Tricross.