For a road bike, small wheels 20" ETRTO 406, 18" ETRTO 355, 17" ETRTO369 or 16" ETRTO349 are not a drawback at all.
With a well designed bike, small wheel provide as good performances than big wheels and provide more fun than big wheels.
The only problem for some sizes is the (very) limited choice of tires.
Using bigger wheels for a road folding bike is a wrong choice, it just makes the bike bigger and heavier. Using 24" wheels ETRTO 507 (or even worse ETRTO 520) for a road bike has, on top if folded size and weight, the major drawback that there are very few road tires available in those sizes.