That sounds sensible other than maybe removing the cleats for walking. That seems like a lot of trouble if you will be taking them off and on much. I never bother to take my cleats off and it has never been a big issue for normal walking around. On the other hand I seldom walk much more than a mile on hard surfaces on tour or 2-3 miles on dirt. Also I don't usually take days off from riding except in special circumstances like your stop on Yakushima.

I will also say this... If you do decide to take your cleats off and on be sure you can repeatably get the placement and angle just right each time. A little bit of error on that can cause knee problems. You may find it to be a non issue if the cleats leave definite marks that make for easy precise repositioning. If there are multiple marks/dents from the cleats on the shoes be sure you know which ones to use.

I do this routinely on my bike tours. I have cycled hundreds of km to get to a trail to hike 30 km or more. I have tried MTB shoes and the Specialized shoes I used were definitely NOT good for hiking and the cleat was dangerous on rocks. I switched back to platform pedals with old metal toe clips and straps. My New Balance trail runners with Superfeet inserts turned out to work fine on the bike as well on on the trail. It's a perfect solution for me, with no noticeable compromise when cycling.

Try A pair of Chrome Kursh ?
they are comfortable & long lasting.
Chrome has a few styles of cycling shoes that work well.

One of the significant issues with most brands of MTB shoes that I have tried walking in over rocky surfaces with smooth or wet finish is that the soles simply don't provide the grip I had hoped for, let alone the cleat. I walk very carefully, placing every foot deliberately if I have to hike over such surfaces with MTB shoes.

A lot of MTB shoes are built as XC racing shoes so less walking is required. The Giro TerraDuros, Empires or Codes all have Vibram outsoles which offer great grip (except on ice but that is expected of most shoes) I will sometimes wear them at work for a while if I don't have time to switch shoes and no grip issues on different surfaces.

I don't use the racing MTB shoes.

I wear trail runners.

Bought a pair of SiDi touring shoes from RBW, when they were last made many years ago .

their molded Hard stiff soles are OK on the bike but not standing on the shop floor..

the type of shoe was in bike shops in Europe 16 years ago, when I passed through, because cycletouring in those places is popular.