Some Ive seen are knurled rather than smooth .. you sure the quill isn't slipping in the fork,
instead of the stem grip around the adapter

There should be some rotational slipping.. keeps from breaking so much
when the bike falls over..