It's just physics and a sliding scale of compromises. All else being equal (eg, tires/geometry), the smaller the wheel:

- the quicker the handling, less stability at speed
- the more jarring the ride, less inherent "suspension"
- the lesser the roll-over capability over obstacles
- the smaller the fold

If all you need to do is store/transport a bike inside a trunk without weather/theft concerns, then a 26" wheeled bike that folds in half can give you a full normal bike experience, including mountain biking. A 20" folds a lot smaller, is squarely in the "suitcase" category of where I'm willing to take my bike, and can do some mild off-roading, but is just a little too big/unwieldy to frequently take "inside" with me and therefore gets lock-up a lot. For me, a 16" bike crosses into the "carry-on" size category for me (particularly if it wheels well while folded like a Brompton), and now I'm willing to take it inside a Starbuck, or grocery store, etc.

So, while the larger wheels can be ridden more places, more comfortably, the smaller wheels can taken more places, more comfortably. Where you want be along that continuum is a very personal decision, but the real answer is, of course, to have a few across the whole spectrum.