You don't really need to wear anything that keeps body heat. Your body makes plenty of waste heat because it's only about 25% efficient. What you do need is to cover as much exposed skin as possible with something windproof. Don't wear cotton or anything that retains moisture. It will start to get wet and make you colder. Look for polyester under many trade names like microfiber, coolmax, ultrasensor, etc. When it's in the mid to high 30's F, I wear an undershirt, jersey and windbreaker, long fingered gloves and leg warmers or cheap tights, and a thin headband that covers my ears. I get a little cold when decsending at 30+ mph, but most of the time I'm fine.
Tom