I keep water bottles (2 of them) in the frame mounted bottle holders, and valuables in a handlebar bag. I do carry a small lightweight backpak that compresses into the size of a fist and usually rides compressed in one of the panniers. I use it mostly when I'm off bike, but I have packed some light and easily crushed groceries in there when my panniers have been full, then rode to the next campsite with it. It's ok for short distances, but based on that experience, I'd never carry anything heavy in my back when touring.
--J