I've had both. There is a difference, but it's not huge. At least not for the start - stop of city riding.

However, there is a much greater range of tires available in 20" size. IMHO, one of the easiest ways to change the way a bike rides is the swap the tires out.

I guess my advice would be that unless you want / need the smaller fold that a 16" wheel bikes provides, a 20 incher is a safer bet.