I would think the "fit" of a bike would vary quite a bit depending on the nature of the use of that bike.

Ditto Kensington's comments above.

Myself, I've got a couple of bikes. One requires a bit of a stretched-out, more forward-leaning riding position, with a much longer (56cm) top tube that's used for commuting. Whereas, the other bike is for a nearly-upright riding position, with a much (50cm) shorter top tube and swept/riser bars and stem that's also intended for urban commuting. Two different riding positions, each bike's format matching the riding style/posture intended, even though both are used on similar roads for commuting. Still, just based on the riding position of each, the formal "fit" is quite different between the two.

Years ago, I had a more-upright postured bike that was completely suited to commuting around town, but it was wholly inappropriate and incapable on the dirt trails of the nearby fields. That riding posture, alone, simply couldn't cope with the inclines, let alone all the other little inefficiencies that bike's format created on those trails. A swap of wheelsets wouldn't hardly have addressed those issues.

Assuming, though, that your riding position and riding "style" (aggressiveness, weight balance, etc) isn't all that different from one activity to the other, then in your case perhaps just a swap of wheels would accomplish what you're looking for. I suppose, then, a single fit could result in a given bike being capable of handling the best compromise possible on that bike.