most of the time I leave my pannier on unless I'm wary of the surroundings then I take the few seconds to take it off. Thankfully most of the places I go are in low crime areas.

I have a pair of Axiom LaSalles' that have an ingenious locking mechanism to the rack.