The off season will be what you make of it. If you don't ride much or do little cycle related training, you will suffer next season if you ride with people who put the time in all winter long.

I'm in Minnesota where the temps are regularly below 20 degrees all winter long and I still get 10-12 hours a week in be it on the trainer, on the road riding fixed or on the trails riding a MTB. This year I've yet to ride the trainer and my lightest week was 9 hours of saddle time. I don't hammer, I just ride easy 130-140HR and believe it or not, I enjoy the heck out of this weather. Night trail riding this time of year is just about the most fun you can have.

Build your base, don't squander it.