I have my wind stories.
How much is too much?
So much that it's dangerous. If it gets to the point where unexpected gusts could blow you into danger (onto the shoulder or into traffic), it is time to pack it in. Anything less than that, and it depends on how much you want to suffer.
My worst experience was with 35 MPH headwinds with 65 MPH gusts ... and in a playa where there was NOTHING to break the wind and slow it down (even small shrubs can really help). That was truly absurd. About all I can say good about that day was that it set a new standard for how bad winds could be. Anything since is a trifling breeze in comparison.