Have you considered that maybe the (loaded) bike, itself, is out of balance and is naturally leaning to the right? You would then have to lean the bike to the left to compensate.

Check your panniers, front and back separately, to ensure balanced loading.