One potential issue of wheel swapping is that the brake shoes may need adjustment if the rim widths are different. A difference of 2mm is probably OK, but from experience, I know that going from 28mm width on the knobby wheels to 22mm on the slick wheels requires a brake shoe adjustment on my MTB (cantilever brakes).