I wear a bag (I have several different kinds) if I will be on and off the bike running errands (other than huge grocery trips) that way all of my flat fix tools and such stay with me automatically. I use panniers for longer/larger hauls.

Often, I use both - sorta. I have a small shoulder bag that fits perfectly into on side of my panniers. It stays there with all of the stuff that I need every day inside of it. The other side of my panniers is for hauling lunch, clothes, small grocery runs etc. When I get to the grocery, I take my shoulder bag out of the pannier and wear it into the store. When I return to the bike with the groceries, I keep wearing the shoulder bag - now I have both sides of the panniers for groceries. Best system I have found yet.