Personally, when it's windy I'm more worried about being hit by flying debris, particularly tree branches (or, entire trees). It's pretty wooded around where I live, so, maybe, gusts > 35 mph or so?
When I'm visiting Rhode Island, there are far fewer trees so it is a bit safer feeling.
As to wind being demoralizing - it is just a question of mindset. Yes, you'll go more slowly and cover less ground. But, (a) you can work on your form (getting aero; pedaling circles, relaxing face and neck, etc.); and (b) you ARE going more slowly, so there's more time to look around and observe the scenery along your route.
I had quite an interesting ride in RI on a very windy day. I was riding directly into a stiff headwind, chewing on my stem, going about 10 mph on flat ground, and it was quite fun. Not from speed, but from battling the wind and seeing things I had not noticed before along the roads.