You also will have to adjust the brakes to each set of rims (alignment to braking surface, any toe-in you may use, etc). May as well swap out the pads, because the wear on each set may not be exactly the same, and you'll want to keep the same pads for the same wheels.