Agree with tsl's post. As a runner I do all the proper layering. However...

I rode into work this morning at -10, and -18 wind chill. At this temperature you need insulation for your feet. I had on riding shoes with neoprene covers. This was not nearly enough, but it's my first ride at this low a temperature. Depending on the length of your ride, you may need something more. I would have been fine for under an hour, but I was out there longer.

Second thing is mittens. Mittens, not gloves. Don't separate your fingers. They help keep each other warm. Even with quite thick mittens, my fingers were still cold. Not as bad as toes, but far from ideal.

Third, cover your face. I had on a thinner balaclava and oversize, clear glasses. Going into the wind the exposed portion of my face hurt from the cold.

This is all new to me too, so I'm learning as I go.