The reason makers strongly suggest limiting spacers above the stem isn't mechanical.

It's medical and legal. They don't want to be blamed if you impale yourself and injure small delicate body parts in a crash.