For physical impact on me, wind. Cross winds require more focus, energy. Riding into a headwind more energy obviously.

For safety, rain is much worse as visibility decreases for you and, more importantly, drivers.