Very true. Unfortunately some of the bear-proof food boxes are not secure against smaller critters. The only time my panniers have been damaged has been when stored in the food lockers at Yosemite. Twice I've had rodents get into my food there and one time the little critter came in right through the side of the pannier.

On most trips I lock my bike to a picnic table or tree and take the panniers into the tent with me since I find that easier than sorting through them to determine what items I might want to have. Bear country trips are an exception.