rocks in head I use pannier but not in town unless I have loads to carry. Even though the weight is lower with panniers it is harder to trackstand because you know have to move your body more to correct the balence. Consider balencing a pool cue on your hand. If you balence it on the tip it is more unstable but when you deviate from the balence point to move the same amount of weight over to send the cue back up right requires less of a moment than if you had the tip up in the air. A tight rope walker with a long pole with weights on the end only has to tweek the pole to gain some balence becuase the torque is length times force. If he has less length or less force he has to make bigger adjustments. Your weight is now closer to your pivit point (and more spred out) so require moore movment to gain the same torque to twist you upright.

Th the reasons you stated above plus catching the panniers on cars when going through gaps is the reason I don't like pannier in town.