I do this with my cross bike. I've swapped between my cross bike and my road bike and had no issues, but due to harsh conditions I got a new wheelset designed for tougher duty that have a wider rim and had to adjust the brakes. I haven't had to touch the rear derailleur even when the cassette gear ranges were dramatically different (12-23 from the roadbike to 11-32 on the cross bike). Both wheelsets were 130mm width and have the same hub (105). If your cross bike has a 132.5 mm dropouts and you switch from a 130 hub to a 135 hub you may have derailleur issues.