Taking off one rest day a week plus a weekend day is a pretty good system. If you keep toughing it out, the fatigued legs will just keep getting worse - trust me, I've been there. Personally, I don't take days off now but I did at one time and it worked out pretty well.