I am not a fan of using a trailer, but drag of the wheel is pretty minimal. I prefer panniers because I find I can get the load down better that way and I don't want to deal with shipping a trailer.

That said even the weight issue can swing the other way depending on which racks and panniers you choose to compare to. Compared to using Surly Nice Racks and Arkel panniers on an LHT, a Bob and a lighter bike will come in lighter.