you don't have to use a power tool.

just use a decent rat tail metal file. It'll grind away the soft aluminum decently fast, but not too fast so there'll never be 'oh crap, my hand slipped' as with a dremel.

that and it won't leave any stress risers if you use one that has fine teeth.