Our bodies aren't symmetrical. You may have one leg longer than the other. You may have a tilted pelvis.
You may need to fiddle with the saddle height millimetre by millimetre to see if that makes a difference. I know of some riders who have their (non-Brooks) saddles angled slightly to one side of straight ahead to compensate for such things. For one leg longer than the other, you might need to look at some solutions ranging from a thicker insole in the shoe on the shorter side, to different crank lengths.