Will it bother you greatly if some of that water and mud you hit get thrown up on the back of your jersey? If so, get fenders. As for whether gravel being thrown up is a nuisance, I don't know, I've ridden small stretches of gravel on my road bike and have never been bothered.

Maybe you should just ride your bike as it is without fenders for a time and find out what your actual conditions cause. And whether any of them are so bad that it cause you to feel need to install fenders.