It's not about absolutes. It's about averages.
Wider tires could be faster, but only on rough terrain. On smooth pavement, narrow tires are faster. Less weight, less rolling resistance, less air resistance. This isn't a matter of opinion. This is tested data. And indeed narrower tires are used in track cycling which have the smoothest surface around. There's never going to be a day when the guys going for the indoor hour record are using 40mm tires. Wonder why?
On a tour the pavement varies, so we have to compromise on tire choice. If you took a diary of all the roads you pass on a typical tour in America, you're going to realize that those 40mm tires helped you go faster 5% of the time and slowed you down 95% of the time. Roads in first world countries are by and large pretty good.
The story would be different if you were going on a gravel dominated tour, but then you're looking at treaded off road tires which aren't available in the thinnest widths anyway, so the discussion is moot.
FYI the whole theory for why wider tires have lower rolling resistance: that forward energy is wasted when bumps kick your narrow tire upward and the energy is dissipated as heat as your jelly body bounces around, that theory works just as well with suspension seatposts and stems. The unsprung weight is lower so it isn't quite as effective, but the majority of the weight is your body anyway, not the bike and luggage. 170lbs vs 60-70lbs. So even if you completely ignore the previous paragraph about road surface, the idea is still the same.
This is the touring board, not racing. When I am on a tour, I am there to enjoy myself, which does not mean being in a race. My light touring bike is fitted with 37mm wide tires, medium touring bike is used with 40mm or 50 mm tires, depending on where I am going. Heavy touring bike which often is a mix of pavement and off-pavement, 57mm tires.
I have never had a pinch flat on any of my touring bikes, even when used with four panniers. Yet I still run front tires at about 65 to 75 percent of the pressure that I use in the rear tires.
Rando bike is fitted with 32mm tires, road bike has 28mm tires. I would consider those bikes for a credit card tour where I have a handlebar bag and a Carradice saddlebag, no panniers.
If you are in a race when on a tour, I can understand your tire choices. There are some former racers on this forum, I think they would agree with you too. If you feel that a wider tire is too non-aerodynamic, you probably cringe when looking at the fenders on my touring bikes and rando bike. But, that is a choice that I made.