I don't think changing the stem will affect your handling as much as you think - except for your own position on the bike. I went from a 140 to a 120 on my old bike (which was still too long) and it wasn't bad at all. Granted, 20mm is pretty small, but stems are cheap enough that you can try it and see what you think before going to a fitting.