So, your fixed gear rides, your spin-ups, your downhill sprints -
Do you think doing them in cold weather is hurting you at all? You know, even with knees covered, seems you're going to have some more stiffness and more risk of injury by doing speed work outdoors as opposed to on the trainer or rollers.
My vote was indoors with the allowance that when I lived in a place with actual winter (NH and VT), my winter speed work consisted of working on ankle strength and going to the field house and doing running 50 meter sprints taking steps no longer than the length of my foot. I rode FG outdoors, but just LSD, down to about 10 deg F.
When I moved to Seattle, where by my standards they don't have winter, I did do specific speed work outdoors and rode rollers if it was raining really f'n hard.
I didn't vote because I think there should one more option... "Both" because I think you need to do outside, real world on the bike training AND on the rollers, ultra high speed spinning that you can't do out doors.