Your brake caliper should have come with two knurled washers. Use them.

If you are using the washers, get a shorter recessed nut at a local shop or as Batboy suggests - cut the one you have now.