For me, I'd have to say wind. Although I've done it, I'm not a fan of "big wind" rides.

I don't mind riding in the rain, as long as it's not a thunderstorm type of downpour. Drizzle to light rain I can deal with.