Yes, face, hands and feet will get much colder in the wind but for the torso and the legs it shouldn't be that much different - unless you're wearing side or back ventilated jacket then the cold wind will force itself inside. Also, since you lose something like 60% (or more?) of body heat through your head and neck I find wearing a balaclava plus a warm skull cap makes me feel warmer overall.
Adam