Coming back to update this with a happy ending, plus as a record for anybody else who may encounter this problem.

I received the FD-6500 today and immediately installed it. Set the height and the angle, then set the limits, then routed and installed a new cable to the left ST-6400. (this was the same way it was routed before FWIW). After all that, I swung the big shift lever all the way into the inside and the chain shifted effortlessly onto the big ring. When I'd been working with the FD-6400, I'd have to put a whole barrel's worth of tension on the cable to get it to even think about shifting. This in addition to very stiff shifter action.

Anyway, moral of the story here is that the FD-6400 is not compatible with STI shifters, at least not the ST-6400 (the numbers are deceiving here). FD-6500 definitely works as it should, but I bet that the FD-6401 works fine as well. I only got the 6500 because the price difference on ebay was negligible. I guess now I'll just throw the FD-6400 in the parts bin, I suppose it would still work with a friction shifter.