I guess temperature would figure into this too. How high do the winds have to be to keep you inside?
For me safe bike handling is a factor - I ride on a lot of roads with heavy/fast traffic and narrow or no shoulders. When gusts go above 30-35mph, it's hard for me to hold a straight line. Wind chill isn't so much of a factor.
It's definitely worse here when the corn is cut - not much is left to break the force of the wind.