If you're cold, wet, and miserable, that's a good time to head straight home (or to work). Clean up, dry off, get warm, and do the longer ride another day.
Which, in a roundabout way, brings up a question I've had for years. I've seen a bunch of detailed training plans. How do people deal with, for instance, riding 57 minutes in Zone 3 when it's cold, wet, and windy?