The actual reduction in speed is less and less as speed increases. (not effort). Because power needed relates to the cube of wind speed. (or more precisely ground speed times wind speed squared)
Going up hill for example, or accelerating from a stop, the speed is lower and the linear effect predominates. 2% isn't that insignificant - but it is less than what you could lose from slow tires, loose clothes or poor positions on the bike.