A Dremel will be the easiest way to go about it but if you choose to file it by hand, get yourself a round file that fits nicely inside where you'll be removing metal - slightly smaller than the actual axle so you have a little room to work.
As Spoonrobot said, use the raised edge as a guide and don't rush it.