Well for the 2012 Hells Gate Hundred I think something like 95% of the field dropped out due to a wind storm, and I looked like this at the finish (my second ride ever on my S-works roubaix...ahhh....memories):
I had to hold on to my bike to keep it from blowing away when I stopped to fish out my swim goggles and mask. Why did I have swim goggles and a face mask with me? Because this isnt the first time I've been caught in a sand storm in death valley. The 2011 Spring Double also would have probably been unridable in the banana, but I can't say for sure. I certainly had trouble staying in my lane from time to time due to cross winds.
Anyway, you'll probably be fine, and as long as you can ride the thing without the fairing it's no biggie.