Not just the weight of the wheels but more importantly the moment of inertia. For two wheels of identical weight, one 700c and the other 20", most of the mass is concentrated at the rim. Therefore the 20" wheel has a much smaller moment of inertia, which is a function of radius squared. Hence faster acceleration. Simple physics.
tl;dr: You're wrong. And you're wrong.
Do the math. If the weight distribution of the wheels is the same (
e.g., the mass is all at the edge of the wheels - not a bad approximation if you drop the mass of the hubs), the radius drops out of the energy equation. So, it's the weight of the wheel, not the size that matters. (Hint -bigger wheel spins more slowly for the same linear speed)
And again (and most importantly), you're not just accelerating a wheel, your accelerating wheels, a bike frame, and a rider with her stuff. So, even if the wheels were lighter, it would hardly matter.
Saying "simple physics" over and over doesn't do anything useful (except possibly convince us about you except how little attention you paid in your high school physics class).