A 60 mile ride at 5mph is pretty slow for a tour. The least athletic tourers usually average 10mph, they just get into position and role along for a couple of hours then rest. Do you need to take lots of rest stops?
Carrying luggage on your back may be OK for short commutes and an occasional longer day ride but doing it every day for 10 hrs will get really old really fast. Look into a seatpost rack. I have carried a backpack on top of the rack but panniers are more secure.