I bike commuted even when the weather looked iffy, down to about 40, and then you never knew what it would be like on the way home. Some days would be pretty cold, and it was always dark. I did that until I retired at 64. Now at 67 I don't go out if it is under 55. And fuhgeddabout the wind or rain. Clouds? Nah, I won't chance it. I have all the clothes I need to ride in the cold, from booties, leg warmers, wind jackets, gloves of various types, beanies. If I need any of those I just skip it now. I'm just not going to do it. Partly this is because being retired I can ride whenever I want, so just wait for the nice days.
As for the NJ weather, I'm in NYC and it has been a real crapshoot this spring, Just when I think I can get 3-4 rides in a week we get another cold spell, like yesterday and today.