For dirt roads, gravel, etc. I think tire width is what you want to consider. I've done plenty of that sort of riding on a road bike with relatively narrow, high-pressure tires (27 x1 1/4 in the old days and 700 x 28cm now.) It's not a big deal. I've usually done it when my ride involved mostly road but with a section of dirt, or when I was exploring and found a new road heading up into the woods that I wanted to check out.

However, if I was carrying a load, like touring, that would change things. I would still consider it, and it probably wouldn't be a big deal, but I'd want a bigger tire, and I'd probably lower the pressure, from around 100 to 60? 70psi?

My LHT has plenty of room for wide tires. My old Nishiki tourer doesn't have quite as much.

If I knew I was going to be spending significant time on dirt, gravel, rough roads, I'd probably take my old Stumpjumper with 26" wheels, and put some semi-knobbies on it.