Reach. By which I mean the smaller size. Imo, reach is the most important dimension, having been put, over and over on 54s, based on my height, when due to my short torso, I needed a 52.

In general, Lengthening a stem is less likely to impact handling negatively. Ymmv, but that's my experience.