When you did your stress test, what was your lactate threshold heart rate (where you move from aerobic to anaerobic)? When you ride, are you regularly exceeding that number? What is your max heart rate? Has your doc considered a coronary CTA (angiogram) to check for calcium blockages? You didn't mention your height, but what is your Body Mass Index (BMI). Probably anything over 15% is adding extra stress to your heart, especially given the kind of exercise you are doing. I am 5'10", 152 lbs and was blessed with bad genetics. Like you, I ride about 7-8k/yr, do three doubles a year. I have a cholesterol problem, so my doc is watching it really closely. I had a CTA done a couple of months ago and my Left Descending Anterior Artery was 50% blocked by calcium. Scared the crap out of me, because I feel nothing. Not much I can do, other than Crestor, Zetia, exercise and a very low fat diet (which I have been doing for years), but my internal medicine/cardiologist/radiologist suggested that I lower my max heart rate down by 10% to reduce the risk of a coronary rupture. My total cholesterol dropped from 220 to 139, LDL from 124 to 53 and HDL from 64/75. I am 64, and my max HR is 170, so I am now down to 153. When I had a stress test on my bike, my lactate threshold was 150 and Vo2Max was 49.8. So, now my max HR and my threshold are tied together. It is tough to keep my HR down, but the difference in riding longevity is considerably increased. Pretty easy to do a century and do it again two days later.
My recommendation...talk to your cardiologist and drop some weight. A lot less stress on your heart.