There's fit and there's handling.

I can make myself fit various sized frames by using odd stems and such, but my actual correct size range is maybe +/- 1 cm in reach, +/- 1 cm in bar height. There's about a 0.5 cm saddle fore/aft range and perhaps 0.5 cm height (pedal to saddle) range.

My bar is about 0.5 cm higher than I prefer. When I went to 2.5 cm higher it was way too high.

My current bars are about as far out as I can use - I raced successfully for a season with a 1 cm shorter reach. I tried a 3 cm shorter reach bar and used a 1 cm longer stem. The 2 cm reduction in reach (i.e. 1 cm shorter than what I once used) was enough to kill the bars for me (since I nixed the idea of having another frame built just for those bars). I'm considering a track stem (I bought one already) that will result in 2 cm decreased reach and raising the bars 1 cm (instead of 2 cm as with the non-track stem). That may bring the bars into acceptable range - although the reach will be 2 cm shorter if the bars are 1 cm lower then it may be acceptable.

On handling - if you go too short a stem you end up with a lot of weight on the rear wheel and not much on the front. This will compromise handling. Having ridden bikes with stems as short as 90 mm and as long as 150 mm, when I spec'ed out a custom frame I aimed for a 120 mm stem. I can go 110 or 130 and not compromise handling too much. 140 is a bit long, 100 is a bit short, and both negatively affect how the bike handles.