It probably takes about as much time to lose fitness as it does to gain fitness.

I've heard six weeks is the minimum time needed to really notice a difference, certainly in muscle mass.