As you bring the bars back towards the steering axis the bike will tend to become more twitchy. When on the hoods, especially with one hand on the bars, your weight in front of the steering axis tends to stabilize things.

Perhaps a higher rise stem will be a better solution rather than a shorter one?