Wheel rims vary in diameter and weight, and of course tires vary in width, tread and pressure. Fat, lower pressure, knobby tires are slower than thin high pressure tires. Also, smaller 20" wheels are often found on upright bikes that aren't very aerodynamic. Thus when comparing people's experience with different wheels, like a mountain bike wheel, vs a road bike wheel, vs a 20" folding bike wheel, much of the difference is due to tire selection or body position, not the rim diameter.

Assuming the tires are similar in rolling resistance and the rider's body is in the same posture, the main difference between large and small diameter wheels will be in the weight of the metal and rubber around the wheel and its effect on acceleration and momentum. Larger wheels are quite a bit heavier, simply because there is more material in the rims, and that weight has to be torqued up to speed, but once it gets going it's harder to stop. This means that larger wheels are slightly harder to accelerate but maintain their speed a bit better (like a flywheel). So on a folding bike with 20" wheels you can accelerate up to speed a tiny bit faster than someone on an otherwise comparable road bike with 700c wheels, but as soon as you start to coast, you will lose speed a bit faster than they do. I think this is what Kabloink was talking about in post #6. When I ride my Bike Friday I notice it's harder to maintain a steady pace than on my other bikes because it's so easy to accelerate, but it slows down noticeably as soon as you ease up. Over a long course, these accelerations and decelerations tend to balance out, and your average speed depends on how hard you work, how much tread touches the road, and your wind resistance, not the diameter of your rim.

Still there might be subtle differences that could be significant over a long ride. The Bike Friday website claims that smaller wheels offer a slight aerodynamic advantage over big ones in fast riding since they have a smaller profile and don't create as much turbulence as larger wheels, and also in a peloton or pace line the smaller wheels allow riders to get an inch or two closer to the people they are drafting.