I prefer to use panniers for bike touring, but I have carried a backpack strapped on top of my rear rack before when bicycling to a trailhead for a backpack trip. Since the OP plans to be backpacking along the way this seems like a reasonable way for him to carry his gear. A trailer would add 10 lbs. or more compared to just using a rear rack. But I'd make sure the rack is a particularly sturdy one. There'll be a greater tendency to sway with all of the load strapped to the top instead of the more usual arrangement with most of the weight in panniers along the sides.