What size rotor are you using ?
Your not using a 180mm adapter with a 185mm rotor are you.

I'd probably just file the required amount off the top of the brake adapter, where the calliper is seated.