I have read that in any crash involving a cyclist and motorist it is the responsibility of the driver to prove that he is not at fault. I would assume it is similar with a pedestrian and a cyclist.
But yes, slow the hell down when passing pedestrians on an MUP. It's frustrating at times when they are wearing earphones and seemingly completely oblivious to anyone else around them. You still gotta avoid them. That's your responsibility as the faster moving one.