I have heard that rear panniers loaded can make a bike act squirrelly, it might be worse depending on the geometry of the bike. You might try rigging up a basket that goes on the front, say get one of those front pannier racks then affix your milk crate to that. Your weight on the back wheel, groceries' weight on the front, it's more equal anyway.
Some messenger types can put a milk crate's worth of schwag in a large messenger bag, that's an option.
Those Dutch bikes sound sooooo cool, but they may not deal with rear loads as well as some other bikes, you might try your present setup on a different bike too.