I agree, wind is the worst. I can deal with most other weather elements, but high sustained winds are brutal.
I find that the wind effect is even multiplied further if you are pushing winter studded tires. It feels like you are riding with one of the brakes rubbing.