↑↑↑ This. ↑↑↑
There's a slight fitness loss that would start at around 7 days of inactivity, which has to do with the number of mitochondria in muscle cells beginning to decrease. But just a couple of days back in the saddle should restore your former strength and endurance. The strength of your heart and circulatory system, and your lung capacity, don't degrade that fast.