I'm new to panniers but just bought a set of Ortlieb Backrollers. Nice design, the way they fit onto the rack quickly and securely.

I don't know what you would have to do to knock off one of these. Certainly more than aggresive riding.