Since I'm putting off doing some household chores, I looked over the geometry specs for the Lexa. Don't know what size you have, but chain stays of 16.1" and sub 39" wheelbase aren't numbers that lend themselves to using panniers. You may be 'canterlevering' your current rack/bag combo too far past the rear axle, which would easily cause the symptoms that are describing....

The FX would give you a CS of 17.5" and a WB of 41.5" (or greater, depending on frame size), which means you'll have little to no issues riding while loaded with your current bag/rack/commuter load.