I feel like muscle goes away faster than fitness. Since moving to Texas I rarely do rides much longer than 6 miles. In two years my legs have lost a lot of tone, but I find if I want to I can still ride all day without losing my breath, which to me is more important.
Contrast to the first time I rode a bike and felt completely dead after only 1 mile - I feel like I must still be relatively fit compared to the general populace.