Need a proper machined spacer, then all is well , you just don't have a quick and dirty kludge.

Or had a father who machined stuff to close tolerances for decades, to see the difference..