I think the biggest problem is to stay warm and yet not sweat too much. Changing layers in the middle of a ride can be a hassle, especially if you have to do it outside. For longer rides I dress up to what I think is adequate (personal experience is the only thing that helps here, really), then try to fine tune my temperature during the ride by varying the effort. If I feel chilly, I'll just ride a little bit faster.
If it is cold (close to 0C or below), you will need an extra jacket to wear during breaks or in case of a mechanical problem. I just rode 60kms in the night at temperatures around 0C, and could not keep comfortably warm during the roadside breaks (this was rural area, no gas stations or truck stops available). I was OK when riding, sweating a bit but not too much. For breaks I added an oversize, warm fleece jacket, drank hot tea, tried to move around a bit, but I still felt cold after a couple of minutes. First kms of riding after a break were never very pleasant.
--J