You'll want to use a washer in the recess of the crown and brake bridge to narrow the opening to fit the pivot bolt.

Back in the day, you could get special shouldered washers for this purpose:



An older, well established bike shop may still have some of these. Otherwise, try to find one that fits inside the recess on the crown and bridge.