It's a public right-of-way. They have a right to walk where they were walking. It has been said that pedestrians "should" walk on the side opposing traffic, but, so far as I know, that does not remove their right to walk on the right side.

What's the problem? Consider them to be slower traffic, just like motorists must deal with cyclists as slower traffic. You slow down, and pass with a safe margin when it is safe to do so.