Bag attachment to the rack have no motion or looseness
once you have them in-place on the rack?

you can gain some stability by moving the mass of the bags forward,
from the steering axis.

I Beefed up the top tube of my next frame in design,
prior one, It had a tail wagging the dog feeling .. normal design,
built with a lighter but oversize round tube set..
when the rear rack was loaded..