I can't speak about panniers because I won't use them--I'm the world's biggest klutz and have already wiped my face off of the pavement when I had an unbalanced bike. I'm not knocking them, just that they're not for me. I know my limits.
I'm using a Mayacycle trailer and absolutely love it. I've taken it on over 1200 miles in the last few months as I train for my tour. It goes up and down hills, through obstacles (Mexican roads are...hmmm...interesting at times), over speed bumps and potholes. It can be a drag going uphill, but I think it'd be just as difficult to power a heavily loaded bike up that same hill. I like that all I have to do is disconnect the attaching fork and I'm done. I honestly don't notice it behind me when I'm moving. It doesn't affect my stopping distance, nor does it "push" me forward. Since this particular trailer is connected with a fork and is only one wheel, there's no real difference in my turning radius. The only disadvantage is that it does add some length to my bike. This has never been an issue, so far, when parking it. It's kind of nice to know that I can take a little extra crap with me if I want. Of course, the bad thing is that I may take too much crap with me because I have more room with a trailer!
For safety, I keep a red blinky light on it at all times, plus one on my bike (will be buying one of those traffic triangles as soon as I hit the States). I've never had any problems w/cars getting too close or not seeing me. Of course, there are always those $%^&* who like to sneak up and honk, but that's a different story...
I think it all boils down to what is comfortable for you. We can all tell you the benefits of panniers vs trailers or trailers vs panniers, but you have to decide what's important for you.