My concern is the dark gray color dust because that is Aluminum oxide. That is the stuff that most grinding wheels for grinding on metal are made out of. When the brake pad surface is saturated with Aluminum oxide, that can eat away at the rims. You can't avoid Aluminum oxide because a fresh Aluminum surface will oxidize in seconds, but you can try to minimize the damage it can do by trying to avoid too much of a buildup of it on the brake pad surface. I don't get obsessive about it, when it looks bad to me or the brakes sound scratchy, that is when I will clean a bit off the pads. In my case that usually means maybe once every three to five hundred miles when it has been dry, more often when wet.