I've toured both ways with my Bike Friday Pocket Rocket and agree with nekohime that a key consideration is how much stuff you plan to carry. Another one is whether you want the flexibility of doing a one-way tour since the trailer/suitcase makes it much easier to fly into one city and back from another.
I found the handling with the trailer to be excellent, with no stability issues even on fast descents. But it still felt a bit odd when going over bumps to have the trailer give a little push or pull at times. The trailer never caused me any concerns on the road. I thought it might in some cities, such as Toronto, with lots of streetcar tracks, but there was never a problem. If anything I got a little more clearance from passing cars due to the trailer width. On my first trip I was concerned about visibility one night when I was still on the road after dark. So I bought some reflective tape and applied it liberally to the side of the suitcase that faces to the rear.
My main issue with the trailer is that it added considerable weight and a bit of rolling resistance. I try to keep my camping gear load down to about 20 lbs. and the trailer/suitcase added significantly to that total. This wouldn't be as much of an issue for someone carrying a heavier load, but for me it makes it preferable to use panniers except on trips where I want the one-way option.
Personally I found it just as easy to unclip the panniers as to undo the trailer hitch - with either arrangement I always had a fully unloaded bike once I reached my campsite. And I found it somewhat easier to deal with carrying the panniers up to hotel rooms than dealing with the trailer. Yes, it's only one piece, but a big and awkward piece.