Two things come to mind as I read this post. One, the body does need rest to adapt to the stress of hard work. Two, what constitutes recovery differs depending on level of fitness. Had the OP not had the level of fitness he did, complete rest may have worked better. But because of his fitness recovery could rightly include modest workloads.
__________________
A conclusion is the place where you got tired of thinking. - S. Wright
Favorite rides in the stable: Indy Fab CJ Ti - Colnago MXL - S-Works Roubaix - Habanero Team Issue - Jamis Eclipse carbon/831