Anything more than a light pair of leg warmers is overkill in SoCal. Try going commando on the knees. See a doc to make sure you don't have something else going on.
If you're older (like me) and have had several arthroscopic surgeries on your knee (like me) and plenty of osteoarthritis... your knee will ache whenever the weather changes, particularly if it gets rainy. There's a reason old people like to retire in Arizona and it's not just the warm weather.