It depends. The quality of a dirt or gravel road can be incredibly variable from place to place and depending on the time of year. Here in New England, we tend to have dirt more than gravel roads (though the distinction between dirt and gravel is kind of murky), and while they will have some holes and washboarding, the better-maintained ones don't tend to be much slower or rougher than the paved roads. They seem to get pretty well-packed and smooth around here. I've also ridden gravel roads in Tennessee and Georgia where the road surface is golf ball-sized chunks of gravel embedded in dirt, and they are definitely more uncomfortable. Though I did ride them on 23mm road tires. Most are perfectly rideable. But I've done at least one road on skinny tires where I had to turn around because it was just wasn't rideable, not in a fun way anyway.

A friend and I did a long weekend trip from western Mass up through southern Vermont and back a number of years ago, and at one point in Vermont the main route to get from one little town to the next was a dirt road, for something like 9 miles. It was really well-maintained and very pleasant to ride on.

That friend and I also did a short section of bike trail in Maine on a trip the year before. That had a crushed gravel surface, and it was pretty terrible. The gravel was loose and pretty deep and it really sapped the energy to try and push through it.

Overall, I really like the remoteness and sense of adventure of riding on dirt and gravel, even if the quality of the ride can be unpredictable. But for really eating up the miles, you're just going to be faster and probably more comfortable on paved roads.