even if the whole thing moves?

it doesnt happen with every brake pull but when it does the whole thing gets shifted to the side, as if its actually too loose