You are correct. In ear head phones can be highly damaging to the ear. Way more than any wind could be. The wind may damage high frequencies in your hearing over a long period of time (e.g. truck drivers who drive with the window open for years. Their hearing gets 'worn-in' you could say in the frequency range of the wind they hear all day.)

An in ear head phone could damage any part(s) of the frequency range in your entire range of hearing. Even at a tolerable level, headphone use for over an hour is not recommended without a break because of the damage it can cause to your eardrum.