I've never had a pannier stolen.
I avoid big cities on my tours, but when I am in a city, I take extra care. When I stop for lunch or go to a grocery store in small towns, I often don't bother to lock my bike. Maybe I am tempting the Fates, but that's what I do, and I have not lost a bike to a thief in 22 years. (I can't recall whether it was locked up, but I do remember it was on the outdoor porch of my house.)