Your intuition is correct. Move the rear panniers as far forward as you can without causing them to clip your heels. Pack all the heavy stuff at the bottom of the panniers and make sure the heavy stuff is at the rear. Even moving the rear panniers an inch forward can make the difference of wobble or no wobble.
The problem isn't with the rear of the bike, it's with the front. And it's not really a wobble. It's that the front wheel wants to pull to one side or the other when I'm going uphill at very slow speeds (below 6mph or so). Moving the weight on the front fork rearward a bit should help because the weight won't be in front of the turning axis of the fork. I tried this change on my commute yesterday and it felt a little better. I'll have to wait a couple weeks until I have time to load up and hit the hills again to see if it helps when fully loaded.
On a side note, yesterday was the first time I've ridden with only front panniers. And I like it. I'm gonna try it for commuting from now on. I could see where it would help in the winter to stabilize the front wheel a bit and keep it from being so squirrelly when you hit a patch of snow.