It depends on the frame, but generally speaking, the bridge between the seat stays serves little/no structural purposes except to offer a place for the rear brake to mount. Many disc brake frames do not even have bracing between the stays.
I would be more concerned about how much material is left surrounding the brake bolt after drilling it out - assuming you remove a lot of material.