This is 100% correct. Do not let anyone tell you otherwise.
Fit is mostly based on the top tube, not the seat tube (although you obviously also need to stand over the bike too). A 55cm Lemond has a 56.5 top tube.

The saddle set back has nothing to do with reach and everything to do with positioning relative to resting on your sit bones and position of your leg anatomy relative to pedal spindle and bottom bracket, etc, etc. Once this is properly placed you get the stem length required for the comfortable reach you want. If you still need a 50mm stem, or even that 70mm you have, the frame is almost certainly too large. But you never never adjust the saddle setback to adjust reach. Way back in my ignorant days I did this after being sold a too large frame by a shop. I spend 2 years of painful hell on that bike until I did a fitting showing me where to start on my setback (a huge 10.4 cm) and then reach.

By the way, my wife is 5-9.5" and rides a 53cm Lemond perfectly and could easily be a little taller and still have excellent proper fit on the frame. But that's not to say achieving the right fit on your frame is impossible, as it may be.