It is definitely the wind against your ears. I cover mine with some specially shaped headphones that I selected because their shape minimizes the noise wind make when it hits them (I also play music through those). They greatly reduce the wind noise, I enjoy the music, keep the volume low, and can hear approaching cars better with the phones than without. I also use a rear view mirror.
On really windy days, I doubt there is much you can do to overcome the wind's effect in covering up the sound of approaching traffic. Depending upon your direction of travel in relation to the wind, it can render approaching traffic virtually inaudible.