I think quite the opposite. Crosswinds are what can cause real trouble for you. A headwind makes it tough to pedal, sure, but a strong gusting crosswind can push you into traffic or off the road.