I can't find a pattern in my own downtime. Sometimes I can go for a pretty long time without riding and not feel a difference. Other times, an extra day off completely undoes me. There must be complicating factors....
I'm not very fit to begin with, but since I never need to ride very far anyway, it all sort of works out.