I usually ride on pavement, but not always. I've toured for long distances on dirt roads with panniers without even thinking about it. I wouldn't do a technical mountain bike trail with panniers, but other than that, I wouldn't hesitate to use panniers on any surface. They do look a bit dorky, true, but a hot, sweaty back is even more dorky, and not having to support the weight on your own body is way easier over any longer distance.