When I'm getting blown backward, I usually call it a day.
Honestly I've never stopped due to wind, but I've wasted plenty of hours peddling against it for nearly no gain. I have a fairly large frame over six feet tall, and I swear my body acts like a sail.
I'm pretty sure that your mountain winds are worse than mine though.