Worse than riding into the wind is riding behind someone constantly breaking wind.
In all seriousness, nothing sucks more for me than 0-2C, damp, windy (35-40km/h) rides, especially when it's hitting head on up all the long hills in my area. Hate those rides with a passion but I still do them because not riding can suck even more.