I use a single pannier. The only handling issue for me is when I get up out of the saddle--but that may just because I'm a bit spastic and have more side-to-side movement than I should.

You might be able to lose a little weight by keeping your lock on the rack at work.