Yeah, you have to give your body time to recuperate. If you don't you're just wearing yourself down instead of building yourself up. I'd cut back the hard rides to twice a week and see how it goes. Go really hard on the hard days and really easy on the easy days. If it turns out you can handle more than that, you can make another adjustment later.