Clothes for XC skiing and cycling are very similar, actually, and I know of a lot of guys who wear their XC clothes cycling and vice-versa.
The thing to remember is that you don't want to overheat and get damp. That'll cause problems for sure. My rule of thumb is that I should feel very slightly chilled before a ride. Actually riding heats you up.