I think if you ride too much you increase the possibility of burn out. I do a lot of cross training in the winter months. I hit the desert with the dirt bikes, go mountaineering, snow skiing, lot of different activities. I still ride some to keep the fitness but it's not a top priority. That way when the serious training for the ultra-distance season starts again I'm fresh and raring to get back on the bike. If you're riding on the trainer a few days a week in the off season you won't loose too much form.
btw, as you are now aware, even during the cycling season you need to give your body recovery time. Whenever you do a big ultra-event you're tearing up your muscles. You need to give yourself some time to repair that.