You've got the right idea but you also need to stuff an allen key into the tube nut in the back and tighten it into the new position.
If it still keeps creeping on you then you may need to strip the pivot down and clean then lightly grease the bushings and plastic bearing liners. Or your housing is creeping and pulling up on the arm side. For the centering tension balance to work the housing cannot creep since it forms a spring that will fight the centering springs of the caliper. Ensure that the housing stays in place and then that the tubular nut is tight and that the arms are not stiff and you'll find braking Nirvana.