Given you only need it tight enough to hold the brake in place, you're being paranoid...

You'd have to crank it a very great deal harder than that to cause any damage; in fact I doubt you could damage the frame that way.

Unlike a typical steel or ally frame, you're not looking at a hollow tube with a little bit of reinforcement; odds are it'll be solid all the way through.