That is probably the main part of your problem.

Get the 3/32 and run a bit tighter tension to make up for the imperfect chainline.

Also, try getting a fixed specific, unramped chainring.