Bouyancy also plays a role in that example (but is related to mass too - but if you could reduce the volume of a pingpong ball enough to match the density of the steel one, it would sink pretty damn fast even with its much lower mass inertia) but yeah I completely agree a lighter rim will slow more quickly.

Also don’t think this has been mentioned, inertia of the bike and rider is 0 when spinning the wheel but high when rolling along the flat, makes the hub bearing friction even more negligible.