If OP has different rim widths, which could be the case since one set is for gravel and one for road, he also would have to adjust rim brakes between wheels. And you have to adjust rim brakes frequently even with the same wheel and rim trueness becomes more important.