Mikey, I agree with Shovel that your expander plug is slipping. I had the same issue. The stem was torqued to spec, but when the plug slipped, it allowed the stem to start sliding up some. I"m not exactly sure how, but it did.

Also, while it is often good to have a spacer, whether its 1, 2, 3 or 5 mm, above the stem, it is not always the case. The instructions to my wife's Tarmac, for example, said to NOT install a spacer above the stem and that doing sou would void the warranty on the fork.