Another simple possibility: Too much weight on the rear, not enough weight on the front. Such an imbalance can cause the front to wander. The corrective action of trail is not able to be put to good use as it could be.
It can be un-nerving to have two totally different bikes depending if the rack/panniers are full or not. The same effect can be caused by rider position. It's the center of gravity being too far aft at fault.
Try moving your seat forward or using your hoods/drops or get a longer stem. Or a combination to move the CG to a more central location on the bike. A loaded front basket would work too if that's how you use your bike anyway and don't want to adjust fit.
With a light front end, you may also be feeling the effect of literally levering the front end around from the torque generated between the seat that is in line with the frame centerline and the contact point on the pedals that are off set from the frame centerline.