Weather is the most consistent challenge for me with regard to touring. Weather conditions can make or break a tour. We have very hot, humid summers in the Southeast, so I do very little touring here in the summer months, unless it's up in the mountains. Rainy days also can make a trip miserable, although I have been amazingly lucky in that regard on my tours. I don't mind cold weather, as long as I have the proper clothing and gear. Wind can make a tour much more difficult but can be dealt with.
Most of my longer tours have been during spring or fall, or farther north if during the summer. Summer weather is generally very pleasant in northern states of the US, assuming you don't get caught in severe thunderstorms or extended rain. When I tour in the South, it's almost always in spring or fall for longer trips.