When they don't work anymore. It's a process that sees them not working as well, having to apply more and more pressure until there is nothing left in the lever.

That is when I replace the pads.

.... or if they get fouled with oil or grease or some other material