Loosen the recessed nut and re-tighten it (without overtightening it) while holding the caliper centered. If it keeps happening, the nut on the front of the caliper may be too tight.

How often do you lube the pivot points? If the answer is never, that's most likely part of your problem.