Do not use the road-specific Speedplays off road. I know from experience.
I was riding a back road, and came upon some road construction that closed the road. The bottom line is that I had to go through the construction, as going back or around was not an option. While walking through the construction I got some dirt in one of my cleats. I spent the next 20-30 minutes digging the dirt out in order to get to the point where I could snap in again.
It takes VERY little dirt to gum up the works.