I had the same size issues. I could ride the Schwinn Madison medium, and I am 5'8 with a 30.5 inseam..but the bar hit my crotch. On the small, I felt more cramped.

I tried the fujis and they rode smaller to where a 54 felt right to me up top but I still had the TT in my crotch by a little. I would rather be comfy when pedaling, so I took the 54 and dealt with the higher TT, which to me is not a huge issue. I also keep the seat a little lower than 5 inches as wll, but that could change since I am still fine tuning it. I plan to get a reccomend on the height from my LBS in a few days just to see what they think.