I love my panniers. I use to ride with a backpack, but I could not stand my back getting all sweaty (especially when going to work).
I do have a Timbuk2 messenger bag, in case I need to haul a little more than my panniers can carry. The messenger bag, if set up right, can be comfortable. I have it set up so the weight sits on the rear side of my hip/lower lower back. I don't sweat nearly as much.
The only thing I don't like about the messenger bag is sometimes, when bunnyhopping or hitting a big bump, the strap will dismount from my shoulder and fall onto my arm. It hasn't caused a wreck as of yet, but my riding is more cautions because of it.