Backpack for me. I travel light, with usually just a shirt and underwear (tools etc in a seat bag). Maybe some lunch or a book. If you're not going to carry more than 3-5lbs you don't need panniers. Once in a while I have to carry my laptop and that is uncomfortable.

I use my backpack for grocery runs, too, but that only works because I live so close to the shops; it'd be too heavy otherwise.

When the sun's shining I ride a road bike which doesn't accept panniers anyway.