As you note, it depends heavily on your situation.

Working at suburban jobs or cities with light traffic, I prefer the Carradice saddle bag (usually with panniers) to keep weight off my back and the ability to carry plenty of groceries or anything else I might get when shopping. I find the bike handles fine (not that 40-50 year old utility bikes are know for aggressive handling) and I don't need to (or want to) weave through traffic aggressively.

When I lived in Philadelphia years ago, I used a backpack, primarily because my city rides were much shorter (2-3 miles each way vs. 5-10 miles) and I didn't have to worry about theft (taking the back pack was more convenient than Carradice bags or some panniers).

I suspect in Philadelphia now I'd use panniers that are easier to attach and remove.