It's not nearly that easy to disturb the front wheel. (atleast not on any bike I've ever ridden).

It might seem like your front wheel and handle bars are just waiting for the slightest imperfection to fling it sideways. This is not how it works. You have the rotating momentum of the rolling wheel and you have the forward momentum of the bike, both of which work with the tires' resistance from the road to keep you going in a straight line. The bike wants to go straight, and the bars/wheel resist being turned.

I run over lots of stuff riding with no hands. It never upsets the front wheel. It would probably take a curb to knock the front wheel sideways. Or slipping into a groove in the road, like at a RR crossing. Basically, something big enough to make you fall anyway, and which you should be able to see coming.

The most likely way to fall while riding with no hands is loss of balance. Remember, you steer at speed by leaning, not so much by turning the handlebars. So if you run over a branch, which upsets your course or your balance sitting up in the saddle, you may lean to correct your course, and then you may overcorrect and fall. That's much easier to do with no hands because your upper body is unsupported. But it's also possible to improve your balance through practice and be able to maintain your balance over bumps.