How do you know you have to slow down to "protect your bike"? If your bike is going to structurally fail from crappy roads, I wouldn't even ride it on good roads.
However, will the fatter tires that a cross bike can clear make the ride more pleasant? Yes.
__________________
Good night...and good luck