> lot of dirt is sticking to the chain.
> Should I be concerned about this?
> What's the best way to clean the chain?

Yuck.
Yes.
Many different ways.

I use the Finish Line chain cleaner and their Citrus BioSolvent. Normally, 2 passes cleans the chain extremely well (I clean every 2 weeks or more frequently if needed). Try to get any add'l grit off of pulleys with spray can version of the BioSolvent. Wipe down.

Wipe down chainring teeth (front and back) and in between cassette cogs (use a non-shredding cotton shop towel, get those cheap from Costco).

Apply (per directions) Rock-n-Roll Gold to lubricate drivetrain. I use White Lightning on the Speedplays, Finish Line Teflon on the other parts, but only Rock-n-Roll Gold on the chain.

Chains are cheap (~$20 for a new DA chain on sale). Chainrings and cassettes are expensive. Keep the drivetrain clean will only prolong the life of the more expensive parts and also keep the bike running smoothly.

I also put a new chain on every year or 4000-5000 miles, whichever comes first.
Every once in a while, it is good to remove the cassette, clean, regrease hub, reassemble.

When the chain and cassette are sparkling like new, the bike just looks ... uh, faster.
Best.