Ever wonder why triathlon bikes tend to have 650c wheels instead of 700c? The 650c wheel will spin up faster at the beginning - less mass to begin rotation - and therefore have a slight advantage at startup.
But there's no free lunch. Drivetrains being equal, 700c wheels have a higher top end than 650c, so you are ultimately gear-limited at the very top end of the gear range with 650c vs 700c assuming gear ratios are equal on both bikes.
Now extrapolate this argument to 20" wheels vs 700c. Same rules apply. They spin up a little faster, but the difference is more obvious. But then we get another variable to deal with, namely, rolling resistance being greater in a smaller wheel than in a larger one.
So if we compare a 20" wheel to a 700c wheel, the 700c wheel has higher top end and less rolling resistance. The 20" wheel spins up faster but has more rolling resistance.
We didn't even mention aerodynamic turbulence/drag yet. The smaller wheel has less drag than a larger wheel, but that difference is probably so miniscule that in real world conditions (i.e. outside world record setting) it's not a factor.
That's the way I understand the physics involved. As always, YMMV, ask your doctor or physicist, etc.