If the pannier is at the rear (rack mounted), it won't make too much of a problem. Riding without hands on the bars will show bike steering to the loaded pannier side, but otherwise, it will be reasonably neutral.