It shouldn't do damage, but it's possible there might be a very slight misalignment which would lead to brake rub. Do you know the business card trick for aligning brake calipers?

The other thing is if the rotors are worn on one wheelset and new on the other, the pads will be less tolerant of misalignment on the latter. Maybe use those wheels to set the caliper alignment, which - again, IDEALLY - should work fine with the other wheels.

Caveat: My experience with this is limited to one bike with two wheelsets, but it was successful, and yeah, the pads rubbed a little on the new rotors till I realigned them.