Ride more frequently than just two or three times a week. Right now you're jerking your body back and forth between building-up and knocking-down. Your between times are long enough your body thinks the need for building-up is past. Then you smack it again.
Personally- I disagree with this- and I do realise that not every "Body" is the same. When I am in training- I do 3 or 4 rides aweek. Long one at the weekend- and after a 2 day rest- a shorter ride that has more effort put into it. Those rest days do have a recovery ride if Stiffness or muscle ache come in- but They have not been necessary this year at all.
But Last year- when I was training for a different ride- I did ride every day for about 3 months. After a month of it- I found that I was getting tired so that was when I started on the "Hard" rides every 2 days- and the in between was just a 20 miler taken at a far lower pace. But after the 3 months- I reverted back to the every other day ride and found the rest days improved my stamina- Climbing ability- and mental state. Speed was not a problem as I rarely use it.