Doesn't bother me. I generally don't like panniers because 1) they are awkward to carry around when you take them off your bike, and 2) it's not easy to get stuff out so I only put stuff in that I won't need for a while. When I'm commuting I only put clothing in it so the pannier is light.
Anyways, have you ever coasted on a bike without actually being on the bike? I mean, the only part of your body on the bike is one foot on a pedal and the rest of your body all on the same side of the bike, with your hands holding the handlebars, of course. That's the way I usually mount my bike as I push off and swing my leg over.
If you can coast a good distance without actually being on your bike, then the imbalance caused by a single pannier should not be a problem.