A lot of old school tourists did it with rear panniers only in the 70's. That used to be how it was done. There can be some handling issues that way though. It also really restricts you weight wise, but that's not necessarily a bad thing.
You might look at a frame bag as well, to allow for a bit more carrying capacity.
Something like this:
http://www.jandd.com/detail.asp?PRODUCT_ID=FFP
As to the trailer and rough terrain? Remember, I pulled a heavy trailer across the fire roads up in Michigan in sandy conditions with a road touring bike.
__________________
. “He who fights with monsters might take care lest he thereby become a monster. And if you gaze for long into an abyss, the abyss gazes also into you.”- Fredrick Nietzsche
"We can judge the heart of a man by his treatment of animals." - Immanuel Kant