Realistically speaking, unless you have some pretty snazzy looking panniers, I doubt most thieves would be interested in stealing them. Rather, they'd be more concerned with what's in those panniers. Indeed, I think a milk crate has a higher probability of being stolen, because most people have a use for them, whereas a pannier is pretty much useless to everyone save a biker. This is provided you're not in crack-head land, in which case everything that's not bolted down is fair game.
I use grocery panniers, which aren't very attractive (a good thing), but hold a lot of stuff. When I'm not out grocery shopping, I only have one attached to the bike at any given time, as I only generally carry a briefcase with me during my commute. So if I get got for the one, I still have a backup until a replacement can be sourced.
If and when I get had for a pannier, I'll begin considering some security options. But realistically, pannier theft is on the bottom of my concerns list.