It's common. Some rims and brake pads do it more than others.

I take the pads out every once in a while and pick the shards out with a small screwdriver. It doesn't take long. You can even leave them in the brake if you are lazy.