Or maybe I can - some loose gravel/sand, with a fat 20" tyre, you could conceivably ride through on the 20" wheel but not the 16" one. So except for a very contrived situation, you're OK 100%.
That's pretty much it. I was on the Confederation Trail outside of Charlottetown in PEI and there was a section where the 16" wheels were manageable, but bigger, fatter tires would have made a small but noticeable difference.