Where I currently ride it is flat, so there are no huge downhill sections where I can become extremely chilled. As such, I just start out wearing a t-shirt and a long sleeved flannel or perhaps sweatshirt, some full fingered rain gloves, my sandals and shorts. Past week it has been about 0-5C in the mornings.
When I have toured under similar temperatures I just wore a wind proof jacket, usually taking it off on the ascent and then wearing it fully zipped on the descent.
I personally would rather bring just a jacket and suffer through a cold descent. Unless you are dealing with truly extreme temperatures, a lot of how you deal with cold is a frame of mind.