There is a bit of a straight answer to it and a bit not....

Usually, the 14", 16", 20", 451 bikes don't have the same materials nor geometry, and all this gets mixed in with the ride experiences we get.

However, I do think that the smaller the wheel gets, the harder it its to negotiate poor roads/trail/gravel.
This can be made up for a bit with wider tires.

I have a 14" Crius smart 3.0 and Java X2. (rather similar geometry and both are alu)
The Crius has 'race' like 1.25" slicks while the Java has 1.75" fatty looking slicks.
The diameter of the wheels become rather different because of this and the ride quality differs too.
The Crius if pumped up hard, is harsh.
The Java feels much better for the same tire pressure, seems to be faster and rides into grass patches better.

Between my 18" Tyrell IVE with 1.5" marathons racer tires and 406 1.25" Raleigh mini velo the experience are as follows.
The wheel diameters are actually only 1" difference if measured with their respective tires. (which may explain why there isn't much of a difference in most ride aspects)
Neither felt more sluggish in terms of rolling (when pumped up to about 85psi), not felt any harsher than the other.
Both had good handling without a clear difference between them.
When tire pressure was dropped to about 65psi, the Tyrell felt more cushy.
The 1.5" tires on 18" wheels do handle packed dirt/gravel ok, I really doubt it on the 1.25" 406 (though I've never really tried much on them)


18" might be a good middle ground? (eg. Birdy; Tyrell IVE)
If not, options for 1.75" or 2" might be a good thing to consider as well.