Joined: Aug 2010
Posts: 12,717
Likes: 2,104
From: Madison, WI
Bikes: 1961 Ideor, 1966 Perfekt 3 Speed AB Hub, 1994 Bridgestone MB-6, 2006 Airnimal Joey, 2009 Thorn Sherpa, 2013 Thorn Nomad MkII, 2015 VO Pass Hunter, 2017 Lynskey Backroad, 2017 Raleigh Gran Prix, 1980s Bianchi Mixte on a trainer. Others are now gone.
I have felt that flex too, but it has been a few years. My knees are old enough and problematic enough that I no longer stand on the pedals to accelerate or climb up hills so I never notice it anymore.
I have always used four panniers, never used two. So, when I experienced that flex, it was with four.
If you pack with the weight in your panniers as low as possible, that may help. I put my tools and heavier stuff in the bottom of the panniers, sleeping bag is the last to go in on top.