For me it's the combination of a few months with only rare bike rides and the fact that muscles don't work as efficiently at colder temperatures. Between 40 degrees and 60 degrees I can pick up as much as a couple of miles per hour in average speed on the same route.

Part of your stamina will return with warmer weather. The rest you will have to regain each year if you let it go every winter.