Try doing an easy recovery ride on those days. You still get out on the bike and you get rest too. In fact since it's an easy day, it's a pleasant ride, as long as you can keep your competitive urges in check and not chase down little girls on "my little pony" bikes when they drop you.