there is a mathematical proof of this. I remember it from a college physics class many years ago, although I don't recall exactly what it was now, after about 20 years of hard riding.

The answer is in the time you take, not the distance you travel. The example given when I was at college. 2 planes - one can fly at 50 mph the other can fly at 100 mph - will a 50 mph wind from point A to point B affect their average speeds for a trip from A to B (100 miles apart) and return.

The 100 mph plane will take will take 40 mins on the trip out and 2 hrs on the trip back. Average speed 75 mph

The 50 mph plane will take will take 1 hr on the trip out but will never get back. Average speed 0 mph.

Sorry the point of this question was - does your flying speed make any difference to the effect of a headwind.

There is the same effect going up and down hills. Since you go slower up the hill you spend much more time going up than you do coming down.