I think you're riding too hard. Get some more base miles in and don't do more than two hard (high heart rate) sessions per week.