Agreed. I've noticed the same thing when I've carried items that increased the effective width of the top tube area of the bike. But individual pedaling styles vary and some riders always keep their knees well out to the sides so presumably they would find it acceptable. Personally I'd much rather have the smaller wheels and no impediments to keeping my knees fairly close together while pedaling.

I do notice some drawbacks to the smaller wheels on my folder. The tires wear out more quickly and tend to be somewhat more expensive due to lessened competition. And, if other factors are kept equal, they have slightly more rolling resistance. Normally this is small enough to be negligible, but I notice that if I let the pressure drop it makes my folder feel much more sluggish than the same pressure drop in the fullsize wheels of my other bikes. But keeping the tires pumped up isn't really a big problem. Smaller wheels also don't negotiate bumps quite as well - leading more small wheel bikes to adopt suspension systems. I haven't found it to be a real issue with 20" wheels, but much smaller and they do start to give a harsher ride.

And there are benefits as well. Despite the OP's claim of wanting more inertia, the universal quest of cyclists and cycle makers is to minimize inertia - both of the bike as a whole, but particularly of the rotating wheels. Since smaller wheels of equal strength can be made lighter, this is an advantage. The smaller wheels also have less frontal area and therefore reduced air resistance. And air resistance is further reduced when riding in a paceline since it's possible for the rider of a small-wheeled bike to get closer to the rider in front.