A lot of people do that - it seems to work fine. You will most likely have to play with packing to figure out what feels best to you - more weight in the panniers or more in the trailer. We are touring with four panniers and the BOB as we need more capacity than the average cycle tourist.