My opinion, based on what you've said, is that probably the 54cm frame will work well. Road Fan's points are right on, and given your seat is slammed all the way forward on the 58cm, and that the 80mm stem is still a bit off feeling, I think dropping down to a 550TT from a 571TT/80 stem combo will work better, my guess with a 110mm stem. The reason being that the shorter wheelbase will put you further back over the BB and make the bike feel more centered, which I think is what Fietsbob was getting at.

i share Road Fan's uncertanty, but it does sound like most of your height is in your legs, so I say go for it! I'm of the opinion that's it's more difficult to fit and more unpleasant to ride a too big frame than a too little one. Also, the semi-compact frame should have a moderate height head tube, so that you'll be able to dial in the right amount of bar drop.