Yes, TT length is the most important part of a fit. Since you didn't explain why, I'll go ahead:

Seat-tube length can be adjusted simply by raising the saddle, to compensate for longer legs relative to a torso. On the contrary, top-tube length can be lengthened / shortened, but with drawbacks. Running a shorter stem can throw the control and steering of the bike off (say going from 100mm to a 80mm). If you have a shorter torso relative to your legs, it's better to drop down a size as you will not be as stretched out as much, or feel like the bike is riding you.

Personally, I'm 5'10 and ride a 54cm because of this very reason. You can always raise the saddle height to make it fit your legs, you can't always shorten the frame to fit your torso.