I've used spacers of maybe 10mm tops but I suspect the issue here is being on gravel roads which will be a heck of a lot rougher riding than anything I've ever done. Lots of variables and tire width and pressure will be a factor also, a big factor in my opinion.

If it was me, I'd want to ride test rides and observe and see how it goes and not bash the bike unnecessarily over rough stuff.

Think it's going to come down to your specific situation and keeping an eye on things.