To me the toughest to dress for is riding in cold, windy, hard rain. Wearing waterproof outer shell to keep the rain off means getting soaked from sweating versus wearing water resistant or not and getting soaked from cold rain. I always chose the former, but eventually end up cold and just have to tough out the last 15 to 20 minutes.
I always carry my rain gear until the summer months, when it's warm enough to just get wet from rain. Heck, some days when it's really hot I wish for rain.