Do the spacers have a shoulder? If so, is the shoulder seated in a corresponding counterbore in the crank arm or chainring?

As a last resort, you can file the spacers thinner.