The only times I have said "I shouldn't have ridden" have been several sub-zero rides (notably one with a 10mph headwind and an actual temp of -9 deg F). About half way through that ride, all I kept thinking of was that I should phone my wife, but the thought of removing a glove to dial the number seemed worse at that moment then just riding on. I made it without getting frostbite, but now I try to limit my sub-zero riding (which can be some of the most enjoyable riding) to routes that are close to home, rather than my "longish" commute to work.