It's the same as an uphill- you spend more time at a lower speed (into the wind, up the hill) than you spend at a higher speed (tailwind, downhill), so your average will be lower.

What's even worse about wind is that a lot of times when it's windy you'll never get that perfect tailwind, so your speed will be lower for the entire ride.

BTW, if average speed is worthless then so are HRM numbers.