You have enough time in the saddle. Two recommendations.
One was touched on by Travelerman, nutrition. Pay attention to nutrition. I get 40 grams of protein after a ride. I use thishttp://www.t-nation.com/store/products/metabolic-drive-low-carb as I like a combination of Whey and Micellar casein:
Biotest Store at T NATION | Metabolic DriveŽ Protein. I found I leaned out too much and I ride less than you. I wanted to keep my muscle mass as much as possible at 60.
The other suggestion is to do something other than cycle. Get a personal trainer and start hitting the weights. If you can squat, deadlift, bench, chin, etc, you will increase your strength.