Are you using an outer ring? If not, can you put this single there?

And removing and replacing the crankset will probably result in pushing it closer to the stay, not what you want.