Pin it.

Use brass pins through the shim, brazed in place. Not going anywhere.

You could also pin the collars and grind off the part of the pin that protrudes past the ID (after brazing).