I would suggest

1. A second opinion
2. Use a heart rate monitor on a long ride, and take the printout from it to the cardiologist for his opinion. Perhaps you can ask "If I keep my heart rate below xxx bpm, can I ride for 8 hours?"

My guess is that no doctor would tell you can't go for a 4 hour brisk hike, so if you could keep your HR around what it would be during hiking, perhaps that would keep the doc happy.