it might be that your cog is eating into the non-threaded part of your hub creating space between your lockring and your cog, allowing it to slip back when you backpedaled.
edit: uppon further review, im not quite sure how this could happen
when you rotafixed it on you were(maybe) just tightening it onto the "threads" you had created, and in the process maybe wrecking your hub. hopefully im wrong though, i didnt totally understand your problem.