Thinking about it from your doctor's perspective. It's likely you are one of the most fit, if not the most fit patient he has in your age group. Imagine that he probably has to nag the vast majority of his patients just to get up out of the chair and take a short walk on occasion.

If he knows you've been riding for many years, it's likely you'll keep doing it and why mess with that?