You're getting passed because other riders are faster than you. Not because they're using backpacks and you're using a pannier.
BTW, I'm in the same boat as you. And I'm a big fan of using a commuting pannier; I don't like backpacks when I ride more than a mile or so.