Are we talking "intentionally" going out on a miserable cold and rainy day? Or do you take long unsupported trips where you have to be prepared.
I ride year-round to the train station, but it's not fun in the super cold or heavy snow (-25F, 8-10 inches are my current records for "crappy"). Waterproof windbreaker and total skin coverage is the key for me. Thermal under clothing is personal preference.
I treat cold weather biking with limits similar to cold weather golf. I prefer: above 40F if it's sunny, above 50F if it's cloudy, above 60F if it's raining, but I've certainly had to make exceptions if I'm on a long trip and the weather goes bad.
And the first essential for rain is fenders, IMO. I spent a cold day in heavy rain last summer without a front fender. The water off the front tire was nicely distributed directly onto my shoes -NON-STOP. Couldn't have had more water in those shoes if I had stepped into a lake. Fenders.