Pick the stuff out, sand the pads off, or replace them. Be especially wary of stone bits that get embedded in the pads -- those things are like a lathe bit to turn down your rims.

I'll note that I've had good luck with Koolstop Salmon pads; they capture much less crud than other brands I've tried, and what does get caught is usually easier to get back out.