FYI, you can get a type of file with a safe edge, used specifically for something like this, where you want to remove material from a surface with no chance of damaging the adjacent surface. They are typically available from a place like Grainger/McMaster Carr (or Amazon, I'm assuming).