I pulled a BOB trailer for 5,000 miles last winter all across the US. There is extra drag that is noticeable, but I attribute it way more to the weight of the stuff I was carrying than to the trailer. The single wheel design works amazingly well. I had about 50 pounds in the trailer and after a short adjustment period, I would completely forget about hauling it. I have used panniers too, and I can say, that properly loaded and set-up, they both work very well.
The comments that you can't feel the weight until the hills have nothing to really do with the trailer, but just the fact that you are carrying extra weight, and you must work harder to get up hills. I don't think trailers are any harder in hilly terrain than racks/panniers. Obviously there is more drag due to the extra wheel, but compared to the weight of you and the bike and your gear, it is negligible.