If I were you, I'd manipulate the profile of the fender and the bracket before I'd start grinding away at the fork. I'd bet that you could get another 1mm clearance easily.