People who think differences in rolling resistance coefficient between different, outwardly similar tires doesn't matter much mainly think that (IMO) because they have no good way to perceive the difference moment by moment. Extra rolling resistance kills you by a thousand cuts. The extra work it makes you do is present on every foot traveled.
Quoting a friend of mine:
One handy way to "envision" the cost of rolling resistance is to compare it to slope. If you go back and look at the power equation, the amount of power needed to overcome rolling resistance is Crr*mass*gravity*speed while the amount of power needed to overcome a change in altitude is mass*gravity*height/time. But height/time is approximately slope*speed. So rolling resistance is Crr*m*g*v and climbing is slope*m*g*v. So Crr scales exactly like slope in terms of power demand.
So a Crr of .005 is exactly equivalent to a slope of .005, or .5%.
Suppose you had a choice of two tires, Crr(A) = .005 and Crr(B) = .004. That's a difference of .001, or equivalent to a change in slope of 0.1%. Over the course of a 1200k randonnee, that's like climbing a 1200m hill.
The difference between a pretty good and sorta-bad tire can be about .002 in Crr. Which is 2x the example given. That's like adding 1,950 feet of climbing to a 300K just due to a poor choice in tires. No thanks. This stuff is hard enough without handicapping myself with crappy tires.