Seems to matter at low speed/ accelerating from a stop, but once in motion, no problems. A couple times this summer I had my grocery panniers evenly weighted on the rear and a fairly typical commuting load on one side of the front rack and steering was sketchy to say the least. I don't want to repeat that experience ever.
More scientifically, if the weight is over the rear wheel which has a more forward moment, the lateral effect is less significant than if an asymmetrical load is applied to the front wheel which has a more extreme lateral moment. I cannot offer more detailed physics than that, sorry.