FWIW, my CAAD9 (with 25c Gatorskins) is horrible on gravel. I can make it down smooth hardpack fine, but many of the roads around here have rocks 2x as wide as my tire, so I have to go < 10 mph to avoid hitting one and wrecking or trashing my bike. It isn't fun riding a stiff, aggressive bike so slow. I'll use it to ride short (~1 mile) sections of gravel I need to connect between paved roads, but that is about it. I also have Speedplay pedals on that bike, and the cleats are notoriously difficult to use after they get filled with mud or dirt, so putting a foot down on gravel/dirt might mean I can't clip back in (though you could always get around that by running MTB pedals - like I have on my gravel bike).
A cross bike with 2 sets of wheels is definitely more versatile than a road bike only, but I think a cross bike still leave something to be desired on the road. I always advocate 2 bikes - one for gravel/rough road riding, and one for flying down tarmac. Unless, you really aren't concerned with having a fast road bike, in which case a more versatile bike with two sets of wheels is completely sufficient.
Last edited by DirtRoadRunner; 04-10-15 at 11:26 AM.