As LowCel has hinted and as you've figured out, it's mainly the competition shoes that are very stiff. This is to combat a number of problems. First and foremost has to do with power transfer efficiency of course. Secondly, SPD style two-bolt MTB cleats are prone to causing hotspots due to the smaller cleat-sole contact area and thus higher contact pressures. Stiffer soles help prevent these hotspots from developing in the foot.
If your're looking for more walkable MTB shoes, try and stick to the line meant for recreational rather than race/competition.