You should speed up and drop the guy behind you at the next hill.
Oh wait, this is not the road bike forum...
Traditionally, it is the overtaking rider's responsibility to make a safe pass. But you cannot rely on the other guy to do what is sensible so instead yell loudly that you have no brakes and the pedestrians will jump out of the way.
Or just slow down a little bit and let the guy behind you pass before the pedestrians.