I think that you start to lose something just about immediately, not a whole lot but noticeable after 10 days. I feel like I've lost a good bit of fitness just going easier this winter and it will take me a couple of weeks of hard work to reverse it.
For me - you may be different - breathing hard from threshold effort, or prolonged effort close to it, in the cold weather causes mucous buildup and irritation in my esophagus, with a persistent cough. Maybe a touch of exercise induced asthma. But going easier it doesn't seem to happen.