Yes and no. Destroy is too strong a word, but dirt can shorten a chain's life. I said can instead of will because only dirt inside the chain counts. What's stuck on the outside doesn't have any effect until/unless it works it's way in.

OTOH, a chain that's dirty and gritty on the outside of the rollers and between the plates increases chainring wear. That's probably what you're hearing. There's no easy answer and every rider finds his own balance between how much dirt he can live with and the time and effort to keep a clean chain.