I live in San Diego. It is never too cold to ride, low 50’s is as cold as it gets during the day.
I can’t imagine riding in 40s or below. Rain is the only thing that stops me from riding.