errantlinguist, There is a nut on each side of the brake stay. Use the nut between the caliper and the brake stay to align the caliper and the nut on the front of the brake stay to tighten. This may take a few attempts, but once right and tightened should stay as adjusted.

Brad