Originally Posted by
carpediemracing
There's fit and there's handling.
...(notes on setup)...
On handling - if you go too short a stem you end up with a lot of weight on the rear wheel and not much on the front. This will compromise handling. Having ridden bikes with stems as short as 90 mm and as long as 150 mm, when I spec'ed out a custom frame I aimed for a 120 mm stem. I can go 110 or 130 and not compromise handling too much. 140 is a bit long, 100 is a bit short, and both negatively affect how the bike handles.
+1
different sizes in the same model will ride and 'feel' different. There are many facets to how a bike ends up feeling under you.
Things like 'Front center' determine a lot of the balance - ie tarmac front-center varies by 8 mm (quite a bit) between 54 & 56.
HT & ST angles also affect that, even though you can/do move the seat around and change stem lengths.
That said there's no way to tell whether 'downsizing' will seem a good move - that's something you need to figure.
labels don;t mean much - a trek 58, from a a few seasons back, dimensionally feels to me like my Spec 56...