I canceled a tour for a slightly sore knee once. The knee wasn't bad, but it was obviously not getting any better with riding. I rode to a nearby small town with a railroad stop and pondered the situation. Finally I decided to take the train home, which meant a 15 km leg of riding at the end. That 15 kms was enough to keep me about 2 weeks completely off saddle before I could ride again. Had I pushed further before deciding to cancel, I'm sure there wouldn't have been much riding for months.
I agree, bike tour is not supposed to be a sufferfest. Just make sure you're fed, hydrated and reasonably rested when you consider your options.
--J