Is this being attached to a round seatstay bridge? If so, you should be using the washers with a curved side to match the bridge and a flat side to match the brake and bolt.

Edit: the washers are shown here in the picture with the caption "Traditional Nut Style" https://www.sheldonbrown.com/calipers.html#mounting