A few thoughts....

Smaller wheel sizes definitely have aerodynamic advantages. That's why they're used for certain types of record events and high-speed recumbents.

However, as noted, they have higher rolling resistance. They're also slightly less comfortable, and the mechanisms used to compensate -- e.g. wider tires, lower PSI, suspension -- all incur a performance cost.

IIRC a bunch of triathletes used 650c tires for the aero advantages, but didn't really stick. I'm guessing the performance advantages were a wash in real-world use.

If you're trying to break the WHPVA hour record, you should be looking at 20" wheels and super-skinny tires. If you're riding on the road, 700c is fine for most people.

I might add that if we were all on 650c, I'm sure some people would be insisting that "everyone" should use either slightly smaller or larger wheels anyway.