I keep my panniers on my bike. Wouldn't go any other way now - except a front rack and bags for longer trips or when I expect to end up with some extra stuff. Picked up a rock about eight inches in diameter six and a half miles from home once, but other than the imbalance in the rear making me more aware going up hills there wasn't a difference that I could really notice.

If I'm just going out for a little exercise I don't bother removing either or both panniers - if I want exercise I can get the same workout in less time, or at least less distance, than if I take things off. For my commute I never know when I'll go out with food, water, and a computer taking up half of one pannier and return, having visited the post office, with two full panniers and a big box strapped to my rear rack. My commute, as it happens, is four miles (so not too long there), with hills (not hugely big or steep) and the prevailing winds tend to be 10-15MPH, usually always from the same direction (tail out, head home.) I've never been able to pin down a big load versus a small load as making me noticeably slower rather than any other reason. I often feel that with a bigger load I am more likely to relax and take an easy pace, though - which I think would have a larger effect than the load would.

Tor