I think it depends on how fit are to start with and what your training load looks like throughout the year. My CTL hovers in the 55-70 range throughout the year and don't really lose anything if I take a week or two off occasionally (which I do rarely BTW.) For me I find it mostly mentally tough to push myself hard during intervals when I've had some time off. If you go from riding 10-15 hours per week in the spring/summer to 5 hours in the "off season" for 3-4 months at a time I would expect much larger swings.