Don't hold the bike because the bike then goes where you're pushing it - which might not be quite
where he's steering it.
Put your hand on his back to assist him in getting started. Don't forget that, at 4, they are still integrating their neuromuscular systems and will be for some time yet. Just keep on encouraging him and he'll soon get the hang of it. I speak from the experience of having taught about 250 kids to ride at our club over the last 10 years, including brothers aged 4 and 6, last Sat.
Once he does, you'll really have something to worry about
Nice pic by the way