A base layer - next to the skin, like underwear - shouldn't really be providing wind protection. It's job is to move moisture outward so your body can regulate its temperature. Your outermost layer (pants) isn't doing its job, if the wind is getting in and carrying your heat away from this area. Look into pants that will keep the draft out without compromising your underwear's ability to wick moisture and keep you comfortable, line your pants with another layer, or consider moving to a less upright position where the breeze won't really reach you there to begin with.