When Moultons were allowed in racing in the early days, they proved almost unbeatable. The smaller wheels allowed closer draughting which conferred an advantage to those using them. They were banned as a result. If smaller wheels were slower then they would not have had much advantage.
Also, in speed record velomobiles, smaller wheels are used from build convenience point of view. If they were really significantly slower, they would not be used.
Great info Jur, I did not know that. It seems the 20" wheels accelerate more easily in sprints also. The combo of better drafting and better acceleration in a sprint puts them off the front in a lot of races.
I have no scientific verification on the 20" wheel accelerating better. Just my feeling and many others have stated the same.
The other advantage to smaller wheels is they are normally stronger.