When the brake stops the wheel from turning, it's what transmits the torque to the frame.

If you left it off, all the force of stopping a wheel would go into the frame through the hub. It would probably not have enough friction and would twist until you loosened the right hand nut and the axle broke.