Bill,
Not sure what the clothing you are wearing is made out of...however from my experience layers with NO COTTON is the way to go. I quite often wear a silk base layer, light wool sweater/shirt and a wind breaker. It is good for me from around 35f to the low 50's range. FWIW I am a low energy rider

I ride to get where I am going with the minimal amount of effort, not to set land speed records

YMMV
This is my concept of cycling...not me BTW.
Aaron