Well, the practical difference between rotating mass vs nonrotating mass is a bit dubious. It's been discussed ad nauseum in the folder forum and my conclusion is that it's very hard to say how much of a difference it actually makes.

Certainly in terms of acceleration a pound off the wheel is worth two pounds off the frame, but the armchair physicists will argue until they're blue in the face that weight makes no difference what-so-ever (rotating or not). However, they are clearly wrong as the "weight doesn't matter" argument applies to carrying furniture just as well as it does bicycles...and we all know that weight does matter when it comes to carrying furniture. The key, of course, is in the physiological losses. Those are what determine just how much more efficient losing rotating mass is compared to non-rotating, but I haven't seen much discussion of those.

Yeah, but the effect is exactly canceled by the fact that the smaller wheels must rotate faster to achieve the same bike speed.

On the contrary. Don't confuse the gross energy with the net energy. Newton's laws may be conservative, but muscles are certainly not. All that body heat generated when cycling is lost energy...and which do you think will work up a greater sweat, riding a light bike or a heavy bike?