I've toured with front and rear panniers, and with rear panniers only. I don't think my bike handles any worse with just rear panniers. I know a lot of people seem to think that distributing the weight evenly between front and rear panniers is the way to go to improve stability. But I always end up bringing lots of extraneous items when I have front panniers on. So just going with the rears helps me to keep my load relatively light.
If you think you might spend a lot of time on crappy roads, or in off-road conditions you might want to consider a trailer. While I have no off-road touring experience, I've heard that it can be better to keep as much weight off of your tires as possible to minimize digging-in in sandy or muddy conditions.