As far as my enjoyment of rides goes, I think wind (especially gusty winds) + ice/snow/rain in the face would usually be at the top of my list, partly because of struggling not being able to see as the wind gusts push you every which way. I tried going for a ride a couple of days ago and started off in >50km/h snowy winds. I didn't dress properly (should have worn a balaclava), and turned around pretty early, but that was mostly due to my chain constantly falling off (I was testing out my winter bike which has no front derailleur at the moment).