When I first started out, just getting a bare bike through a doorway (especially those with a stiff return spring or automatic sliders) or up or down stairs was quite a challenge. It's a bit more of a challenge with bags on.

A bag laden bike is also less wieldy (hahaha) when hanging on hooks on the train or loading onto bus racks, car racks, trunk of the taxi, etc...

Panniers are a pain for multi-stop trips, also. Since I have my bags on today, I'm only stopping at one store. If I had my backpack, I could happily hit up the bank, Target, grocery, Deli, video store all on the way home, not so with the panniers.

The only backpack manageability hit I can think of is in crowds. Squeezing through the throngs in a rush hour train pretty much has to be done with your pack held in front of you.