I turned a one-piece bearing cap + spacer for my wife's bike out of Delrin-- and not even real delrin, but generic acetal homopolymer from McMaster-- and it's been on the bike for over 5 years.

Delrin is outstandingly strong, and should (and does) serve perfectly well as stem spacer material. I mean c'mon, I have some FSA branded spacers here made of acrylic.