yes, low-quality pads will affect it slightly, but it happens with all rims.
during this time of year we're doing a lot of complete overhauls on high-end bikes and every one involves taking off the brake pads, picking the tiny specks of aluminum out of them, cleaning them, then sanding them even (this is all if they're worth keeping).
just keep your rims and pads clean and inspect both every once in a while. not much more that you can do.