A shorter stem will also make the bike handle faster, or in another word ,feel twitchier. ( Smaller turning arc for the same amount of turn.

At faster speeds this becomes more noticeable. As others have said, put some miles in with your setup and relax.