Not if you don't do up your mounting bolts needlessly tight. And don't talk to me about star washers... geez, leave em off. It doesn't take that much to hold your brake in place. If you can move the brakes but you have to mean it, that's about right. The only force acting to push it off-centre is via the cable housing.

I have a feeling the bit the screw goes through might act as the fulcrum then...