fatigue happens at a different scale than your pictures. There is some deformation, but you aren't going to see it from here. A typical fatigue crack will close right up and be barely visible for most of its life.

If you look into fatigue failures more, you will find that the far field stress does not have to exceed the yield stress. That's why fatigue failures are so common, people worry about exceeding yield stress, but don't think about the cycles that the structure will see. A simple representation of this is the Goodman diagram. It's not a perfect representation, but it does show the trade-off between cycles and stress leading to fatigue failure. Actually, the S/N curve is better, even with all its flaws. Only in the case of zero cycle failures will you see far field stress exceeding yield.

I have seen the argument made that thicker straight gauge will crack at the ends because the center of the tube doesn't deform as much. I have never really bought into this theory though. I suspect the ovalizing process or some other part of the process caused this to happen. And since you have never had the butted downtube tested, this doesn't really provide a data point either way. It's possible that your brazing process will not withstand this test.