I use sweats as well. They work well alone down to freezing. Lower than that I wear bike shorts and soccer socks underneath, leaving just the knees with a single layer. I've done that down to about 20°.

This works OK in a light rain. If I splurged on rain pants I'd fare better in a driving rain...