Not all dirt is created equal. Soil is different than sewage. I'm not a prude when it comes to dirty hands. Plumbing is my family's business, and we were seldom sick. But I still wash my hands if I'm flossing my teeth after I use the toilet, for instance.
Handwashing has its place, like preventing fecal-mouth transfer of nasty stuff like hepatitis in restaurants, hospitals, etc.
I'm amazed at how many myths surround something as simple as handwashing. Some very intelligent people I know believe regular hand soap and warm water kills "germs". Mom said so.