I’m not a fitter, but for me everything revolves around bottom bracket to saddle position to bars. In theory I would like to be balanced so my weight is not too forward or set back too far.
Going to a smaller frame for me means more setback in a seat post and, ideally, a corresponding amount longer stem. But in reality getting close after getting the saddle where I like it is probably fine.
For those so inclined, bottom bracket does take into consideration crank length to some degree.
John