If you ride in the rain, crud is going to build up everywhere. The jockey wheels seem to be particularly bad because the chain packs it on tighter. I don't know if I believe that the lube has anything to do with this. I scrape it off once in a while with a rag and fingertip, and I can definitely tell a huge difference in efficiency after I clean it. I doubt it does any damage unless you let it get ridiculous.