I think it is the stem. I just had the same problem with a Nitto Technomic stem. Pain in the rump, but it is just ever-so-slightly too big.

I put some time into sand paper and then emory cloth. Use some heavy tape to protect the area that will show from scratches. Messy, but it finally worked.

Once I had it down, I could see where the scoring showed from each effort to insert. That made the sanding process a bit more focused. Annoying, but it's the cheap Nitto.