Yes, you're right to be worried, wind can be a major hazard if its strong or gusty.

Australian professional rider Paul Crake, who is also the five time Empire State Building (NYC) stair running champion was blown off the road with four other riders by a gust of wind during a road race in New Zealand last year. Paul suffered spinal injuries because of a freak impact with a post or other object. He was a champion athlete and an awesome rider. I was chatting with him during a ride the previous weekend in Canberra (Aust) and its hard to grasp that something so freaky can happen like that.

http://www.cycling.org.au/Content/Na...sage_to_Pa.htm
http://www.news.com.au/story/0,23599,20773293-2,00.html

If its blowing hard or there are gusts from the side be extra careful.