Yes, that is the limiting factor (without going to some geeky overly extended stem).
On my 63cm Cannondales I have to use some Salsa stems that are nearly vertical in rise to bring the bars up. I've got two of 'em (one on the tandem and one one the 3.0 frame). I've never seen 'em elsewhere but they are much much more stable (rock solid) as compared to a Nitto Technomics. However, the "look" is frankenbike. No bike should look like that. Even though they look completely 'geeky' the bars are still below the saddle height, but its just so...wrong.
In comparison I just love to look at the 27" C'dale now (68.5cm). It justs looks like a real bike, albeit with little clown wheels.