The only times I don't like it it when it is so high that all sorts of things are blowing across the roads and crosswinds. Tree branches, limbs, trash cans, dust and dirt etc. Getting those things caught up in your bike is no fun nor is having you and your bike suddenly get shoved over to the right or left by a couple of feet by a sudden crosswind.
Other than that it doesn't seem to bother me any more than when playing golf or even fishing when the wind is up.