It bothers you in the sense that it's "exhausting" to the point that you want to address it. I don't see how it could contribute to physical exhaustion. Are you talking pyschologically exhausting? If so, how does that differ materially from being bothersome?

And I would like to see some support for the claim that exposure to "constant noise levels" of the type created by wind while cycling are bad for hearing. Also, how often are you going to be riding 100 miles/day?