I think a few folks are misreading the OP. He has a spacer above the stem.

OP: as long as there is enough steer tube sticking into the spacer to keep it in place, you are fine.