I agree completely, I personally dont get why you would want your tourer set up very much differently than a bike you ride unloaded.
I think that the reason most people set up a tourer differently from their everyday road/training/fast bike is that they want the tourer to be more comfortable for longer periods of time. The regular road bike is meant to be faster for shorter periods, perhaps with some sacrifice for long distance comfort.
I would imagine that the bottom line is this: the same body position that you want to be in when you are trying to go fast , and consistently pedaling hard, is probably not the same position that you want to be in when you are on the bike for most of the day, day after day.