With the rear rack (especially the JandD Expedition, which is 3" longer than std racks) you're now carrying weight behind the rear axle. The rear wheel acts as a fulcrum, the rack and frame act as a lever, and weight you add to the panniers will unweight the bike in front of the fulcrum. Since this is a road bike, you've probably got the panniers mounted all the way back for heel clearance, so any cargo is producing maximum leverage.
Although this may not seem like much force to counter your body weight, consider your position/posture on the bike is placing most of the weight on the rear wheel, yielding a weight distribution of at least 40:60 F:R, and possibly as much as 20:80 F:R. Your relatively high handlebar position puts your weight further back. You can actually measure the weight distribution with 2 bathroom scales and a helper to read the scale while you maintain riding position.
On some of my bikes, with real skinny tires mounted, I can induce temporary shimmy by just sliding all the way to the back of the saddle. These bikes also have setback seatposts, and no rear rack or any bags.
So, adding that long rear rack and a load is what's causing your shimmy, as you've already surmised. Lots of folks, when they get shimmy, start looking at their headset, handlebars, front wheel and tire, while ignoring their rack and load, which makes for entertaining content in the forums.
A bike frame with long chainstays would help reduce shimmy. You'd be able to locate the load more forward, without your heels striking the panniers. A frame with longer wheel base would help a little too. Some panniers are very square or rectangular, so that you have to push them back so you don't hit them. Others have thoughtfully "cut off" the lower forward corner to eliminate this conflict. Take a look at your panniers, they may be worsening the situation.
A proper touring frameset could help you with shimmy. Longer chainstays plus the capability to load the front of the bike should solve the problem.
As a lower cost measure, you could also try relocating the load to the front wheel via rack/panniers, as suggested by SlimRider.
I learned this lesson once hauling a big cube of bricks in a pickup truck, which shimmied like mad for the 2 mile trip home. The bricks should have been pushed forward all the way to the cab, instead of centered over the rear axle, but the fork truck didn't have enough reach.