Someone already mentioned this but wearing summer lycra around the crotch area in 35 degree weather doesn't cut it. A thicker "roubaix" material works. Bib knickers made with the thicker material works great because it keeps the lower core warm. I wear them down to about 35 degrees (racing), 45 degrees (training).

If you have non-bib knickers they're probably thin material. Wear them under (under!) your regular shorts. This gets two layers in the "shorts" area, one layer on the knees.

I wear booties under 55 degrees pretty much all the time. This means that I wear them with shorts down to 50 degrees (racing) or knickers down to 35 degrees. It's much more important to keep the feet relatively warm.

I'm thinking of converting some of my older bib knickers (roubaix material) into shorts. This way I can wear knee warmers with them.

Basically the more time you spend in that 45-55 degree weather the more you can use knickers. I ride about 4-5 weeks in the fall and maybe 7-8 weeks in the spring using knickers consistently (less time in fall because I dramatically reduce riding time). Around here temps don't change dramatically from AM to noon to PM, and I usually ride in a 1-3 hour window where the temperature might change by 5-10 degrees.

In SoCal I've used knee warmers, but if it's cold enough for them it's usually cold enough for knickers. Maybe an early morning departure time to assault Palomar (55 deg depart), 40 deg or lower at the top, then 55 deg or so when I get back. Valley might have hit 75-80 in the mean time but I've been elsewhere during that time.