+1

The overwhelming majority of cantis and v-brakes don't move against the studs/bosses. They move around a bushing that's mounted in the arm.
On assembly the bushing gets pinched in place by the screw and stays stationary, then the arm move around the bushing. A close fit on the stud/boss is not required.
Likewise a close fit on the stud/boss won't help if there's excessive play between the bushing and the arm. Replace the brakes and try again.