You didn't hear the narration which said something like, "We forgo the ruins this time and instead have lunch here." If you check on the channel and also the narration mentions we have camped there twice and visited the ruins at least the first time we went there. There is also a brief explanation in that first vid about why it is supposed to be one of the most haunted places in Japan.