Brakes are about the toughest parts on a bike. This is because maximum braking forces are much higher than anything a rider can generate for accelerating or climbing. It's also because a brake failure is likely to lead to injury and possible lawsuits, so makers err on the side of safety.

I'm not saying you should ignore dirt, especially because embedded grit can score rims, but don't bother taking them apart to clean mud off. Use a garden hose, or an old windex bottle and spray clean and rinse and you're good to go.

If grit does get embedded in the shoes, remove the wheel, (the brake should release open so you don't need to remove the shoes) and use an old file or rasp to clean the surface of the shoes while they're still mounted.

If you look at older bikes still on the road, you'll find that the brakes are probably the oldest original equipment (other than cables and shoes).