Do you walk at a cardio or aerobic level? Sure it's going to have some difference it makes on you but for that period of time it's not the end of cycling for you. You'll catch back up quick enough.

When I walk with my wife at what is a fast pace for her, I never get over 90 bpm. Even when we were walking 80 miles a week.