My 2 cents.
Your a taller rider on a bigger frame. The stem looks like it could use at leat 20mm.
This would slow your steering down. I built a Nishiki Continental with a 70mm stem and it ruined the bikes handling. Twitchy even unloaded.
I swapped a 110 back on and the problem was gone. From my experience a short stem on a bigger bike leads to twitchy handling.