Front or rear rack? Rear rack with weight and steel frame adds up to flex that you have to control with your wrists if you climb out of the saddle. Put those same panniers on LowRIders and you wll find climbing out of the saddle is only harder because you are carrying more weight. Now, the frame itself may simply be flexing because that is what steel frames do (depending on tube thickness, your weight and riding style (and to a lesser extent, the frame geometry and details),

Ben