No personal experience (knock on wood) but with a wife in cardiology research, I'm interested.
I think there was a cluster of Afib cases in athletes 5-6 years back. Most of those were middle-aged or older triathletes or competitive Masters cyclists. It sounds to me like your cardiologist isn't particularly active, and is lumping you with those cases by implicit similarity. After all, a 24 mile casual ride is about the same distance as a 25 mile time trial, so anything that affects the latter must be causing similar symptoms in the former. Right?
I've discussed my cycling habits with three doctors in one cardiology practice. Doctor 1: Riding on the roads? That's dangerous! Doctor 2: I used to ride mountain bikes a lot, but I broke so many bones I don't any more. Doctor 3: Really? You ride bikes? Cool! That probably is why you're alive today, the secondary vascularization your cycling developed saved your life when you had your heart attack.
Dr. #3 is my cardiologist, right or wrong. You might see if you can find a doctor who exercises himself (or herself).