In my experience, a two wheeled trailer is vastly more stable than a single wheel trailer. I'd be going for a two wheel trailer with a decent wheel size (ie 16 or 20 inch). I think the downsides over panniers are twofold - more drag with more wheels on the road, and having to take and store the trailer.
But I'd still do it this way.