On smooth pavement, the harder you pump, the faster you go. There's no question. On less than smooth pavement it's more complicated. Nothing to do with the rearward force from bumps because for every rearward force there's a forward force from the other side of a random bump. Rough roller test show that resistance decreases with increasing pressure just the same as it does on smooth rollers. It's the upward movement of the bike that uses energy, so to get an increase in resistance there has to be enough upward movement to counteract the inherent lower resistance of higher pressure. That's going to be different with every tire and rider. The other thing that happens is that some tire/wheel/frame combination damp or eliminate lesser bumps before they reach the rider, preventing vibration from contributing to tiredness. Other combinations don't. So that's going to vary with the equipment, too.

The only way to sort it is to experiment with power and speed instruments with your bike on various courses. Otherwise it's all just internet noise. FWIW, I've done all my fastest long distance events on a carbon frame running light 23mm tires at 140 lbs., some smooth pavement, some chipseal. 150-160 lbs. TT riders and long distance racers report similar results. Frame makes a big difference. I run <100 lbs. on my aluminum CAAD9.