Took a closer look tonight, turns out the screw hole that the braze-on bolt screwed into was helicoiled. Was that way unknown to me when I bought the bike used off CL. (Not that that's bad)
Is it possible this is not my fault? Does helicoil fail catastrophically like that sometimes? Or only when the bolt is left loose?