From what you say, it seems to be small gouges near the outside perimeter of the rim. If that's the case, you can file whatever protrudes in the path of the brake pads and you will be OK.
As you file the rim, also make sure you have a smooth surface at the outside perimeter of the rim. Otherwise, the tire may wear out prematurely and blow out at that place.
You might also look at the path travelled by your brake pads. Depending on their width and the width of the rim, you may be able to lower the brake pads a bit. Just make sure they don't rub on the tire, because... Boom!