if you divulge your weight and/or wattage output, we can probably tell you if its your frame that's noodly, or your legs that are beastly. or perhaps both. "stronger" is relative. watts are absolute.

otherwise, your wheels are probably flexing a bit too, even if you build them up yourself. even the strongest of wheels are going to flex some amount, and when you run your brakes with such ludicrously tight tolerances, that's going to be a consequence (between any frame flex, and wheel flex).

you have a few options:
1) loosen the brakes by a mm or two, and deal with brakes that are looser than your delicate sensibilities would prefer. (see rule number 5)
2) deal with the brake rub (also see rule number 5)
3) ride your other bike
4) start engineering better carbon fiber frames yourself, in your spare time.