Originally Posted by Iride01

Is your wheel true? If not there is your problem. Also, if your spokes aren't at the correct tension, then your rim might be staying somewhat true but anyone's guess where they will be while in motion with a load on them.

There is an adjustment screw on those brakes to shift where they center. It's the little 3mm hex that is described in instruction number 1 on page 34 of this:
https://www.servicearchive.sram.com/...nual-rev-a.pdf
might not be exactly the same as your year model but it's similar.