A few general comments:
1. The biggest item is predictability. If I know the road is gravel, then I'll take the appropriate bike/tires and that can be as pleasant as paved roads. What isn't as fun is expecting pavement and then having multiple tens of miles of gravel.
2. There can be a pretty big variation in gravel roads. Some when they are wet (some soak in, others become soft mud) but even when dry, there can be loose coarse stuff graded over vs. relatively smooth. Similarly, roads can accumulate a lot of corrugations and be somewhat of a pain to ride. The expectations thing is back at play here, but having relatively smooth, not muddy, uncorrugated gravel isn't really that different from pavement. The others are ok too if I'm prepared for them.
3. More than about 8-10%, I'll occasionally end up walking, particularly loaded. I tend to be a bit cautious, so descending on gravel is a lot worse than climbing on gravel.
4. When it comes to a busier/flatter road that is more direct vs. a quieter/hillier road that is longer - I'll bias towards the more direct roads.
+1
1. Yes, if I know the route is going to be gravel, then I'll prepare for it, or avoid it. Going back to my example of the 100 km route Rowan and I rode last year, I knew there was going to be a bit of gravel. I was OK with it for the first 2 or 3 km, even up to 5 km, but on the second stretch when it just kept going and going, I was less impressed.
2. You are absolutely right about different kinds of gravel.
In Manitoba and parts of Alberta, you wouldn't even consider riding a gravel road when they've freshly re-rocked it. In spring, they take a gravel road which has finally flatted out and has become a possibility for cycling, they dig it all up, and then they lay down a thick layer of small boulders. The rocks are anywhere from golf balls to hardballs in size ... and sharp. I flattened a wide, thick knobby mountain bike tire on those rocks in attempt to ride one of those gravel roads.
Cars drive over it, the weather pounds it, and ever-so-slowly the rocks work their way into the dirt, or shoot off the side into the ditch. Which, incidentally, creates another hazard for the cyclist. There might not be many vehicles, but if there is one, you've got to take evasive action lest you find yourself in a hail storm of these rocks.
Eventually, portions of these gravel roads flatten out enough that you might be able to ride in the tire tracks ... and that's when the Powers-That-Be decide it's time to re-rock the road again.
When it was suggested we ride a gravel road the first time I came to Australia, that's what was firmly lodge in my mind ... but the gravel roads here aren't that bad. They generally use a finer, sandier gravel and seem to pack it down more. Some sections aren't much different to riding on the road. But traffic and weather can create the washboard/corrugated effect, or wash out bits and pieces.
So yes ... it does depend what the gravel road is like. They aren't all created equally.