Yes things like trail and such come into this but if the wheels are not moving then bike or the wheel is unstable. If small wheels on a bike or off are moving at the same speed as a large wheel then there rotational speed is proportionally greater. Quite often in physics we find formulas for movement involve an mv^2 term, m for mass and v for velocity often squared. I think as the tires slow toward zero speed the greater rotational speed of the small wheels makes it more stable.
The observation came from riding my Brompton versus riding my larger tire bikes.