Not sure what you mean by brake pad marks.
If you're referring to rubber residue left on the rim, this can be removed with scotch-brite pad (or green non-metal scouring pad generally), perhaps soften up with a rag + rubbing alcohol first.
If you're referring to how the brake pads have worn down the rim (friction does grind aluminum away over time), why would you want to grind more rim away?
Sandpaper is bad unless you have scratches perpendicular to the rim that need smoothing. No need to grind the aluminum thinner. If rims don't fail from eyelet cracking or crashes, they eventually fail when the brake-pad-contact surface gets too thin from pads wearing rim down over time. Scotch-brite pad works well to smooth and remove "foreign material" (no WMD's, sorry) stuck to the rim surface.