That is why they make gravel bikes. Basically fast, light, Drop bar road bikes with just enough extra tire, and perhaps a geometry tweaked for more control and speed on a loose surface.

I thought we needed wide tires for gravel and dirt roads, Until I rode with a friend who's hybrid with 35mm tires did an excellent job. So now I understand where gravel bikes come in.

Those mountain bikes with their fat tires (54mm on one I had) are great slugging through water logged grass where my tire sinks into the ground over the rim, Or for climbing a hill with soft gravel and loose stone, But on many dirt or gravel roads aren't necessary. And on the street it felt like it was fighting me. Perhaps this explains the popularity of hybrid bikes.