I think the fact that you as a rider have mass and therefore inertia would compensate for the lack of rotational weight in the wheels. The bike would still be structurally holding you up so it might be a bit unwieldy when not riding (you would need to carry it as opposed rolling it with one hand) but when riding I think it would just be the same as if you had lost 15-20 pounds. You might get some extra vibration descending since there would be no mass in the frame to stabalize vibrations.
__________________
Sunrise saturday,
I was biking the backroads,
lost in the moment.