1. Recovery time (as in sometimes you just need more than a day or even two), 2. cadence, 3. you're not going to make huge improvements in a short amount of time.
Last summer, I rode my bike across Canada. 74 days in total. I live in the prairies, and even though I put on 1500+ kms pre tour, I was not ready for Mountains or even bigger hills. I was one of the slowest at the beginning of the tour, and I remained one of the slowest at the end of the tour. We all got stronger together. BUT, I am way better now than I was last June at the start! My simple point is it takes time to increase average speed. You might also consider different roads, and same distances, rather than travelling the same roads. Monotony is a killer. Cadence is critical, as that is the only real "tool" that you can use on the rode to kick you in the butt. Recovery time. Same as weight lifting. It used to be- Upper body one day, lower body next day. Usually, that works, but not always, and you must listen to your body. The other part of recovery is, using a heart monitor when you're exercising. If you are an average guy like me, when you are exercising, you should get your heart rate to about 140 and keep it there while you're working out/riding. If you're struggling to keep it there, or get it there, chances are that you are not sufficiently recovered. STOP! This is where you have to listen to your body. Take another day off. I have had times where I was 4 days in rest mode. (not very often, but it happens) But come the 5th day, look out! Suddenly someone lit a fire under yer saddle! Quit beating yourself up!