It's normal. Rims will wear down, although very, very slowly over time. The tiny fleck of metal you see are coming from the rim. Nothing to worry about, though you can sand them out carefully if you braking performance starts to go south.