Stems vary in height, but not by much. So do spacers. You can lower the stem by rearranging the spacers like you suggest.