Don't fret your days off. That's the time when your body is rebuilding itself and making itself stronger for the next time. Until you get stronger make sure that if you ride on the off days it's very easy. What the roadies would call a recovery day. If you push yourself everyday without a break you will slow down your development.