Not exclusively, no. On one of my regular routes, I have a 1.5-2 mile stretch of canal tow path that is dirt/gravel with large rocks to negotiate. I will say that riding this stretch on my MTB with my cheap Avenir panniers led me to seek out higher quality hardware for them, after two hooks loosened completely in as many weeks. Although I had no problems with the hardware on my regular commuter rig, I decided the tougher hardware was worth it, in the long run.

I would have no hesitation running panniers over rough stuff, though I'd be careful to inspect them regularly, lest some part fail prematurely. I do think the constancy of vibration is tougher than the occasional (or even frequent) pothole, pavement irregularity or curb hop, though this is purely based on anecdotal evidence and no science :-)