Like recursive, I ride every day. If I feel like I need a day off (very rarely) I do just my commute, slowly, and no other riding. That said, you shouldn't fret about taking an occasional day off. On the other hand, the more you ride, the sooner you will develop the strength and stamina you desire. If you want to ride both ways to work, make it a goal and work diligently toward it. You will get there in no time.