This is from the chainring not being exactly centered around the centerline of the bottom bracket. (Most cranksets out there have this issue to some degree. The best velodrome cranksets are very good. Cranksets intended for road use on derailleur bikes less so since with a derailleur with its spring -loaded cage, it does not matter at all.
Cranksets intended for single speed/fix gear should be pretty decent as far as getting all the machining to be accurate dimension-wise, but there still is you-get-what-you -pay-for. Accurate maching costs time and money. Now, you can improve most cranksets by going through a routine to adjust any slop in the chainring bolts and holes to improve the location of the chainrings. Go to Shelden Brown's website. He describes the procedure very well. (I don't have the link but maybe someone here can find it fast.)
The nuts and washers at the hub have little to do with what you are seeing. Best you can do is adjust the chain tension so it never goes tight at its tightest and still has little enough slack that it doesn't fall off. (I know, your bike will now flunk the "perfect, oh-so-tight chain that it seems every Portland hipster has to have. But if you to Portland's velodrome where the really fast guys race, you will not see any tight chains. Not one. Yes, on the road you want to be a touch tighter than on the track because the track does on t have bumps, but that is all the tighter you need.)
So, if you can find that magic spot where the chain stays on and never goes tight, you're there. If not, try Shelden Brown's approach. If that doesn't work, upgrade the crankset. The Sugino 75 is the gold standard but you will pay. You will be able to get any decent Sugino crankset, even quite cheap road, to work using what I shared above.
I have the 75 on my best fix gear and use quality Sugino and Campagnolo rings. Setting the chain tension is a joy. My workhorse fix gear and my retrofitted Peter Mooney use old Sugino GT and AT 110 bcd medium value road cranksets. I have to do as above but they work very well. Just 30 seconds more time to dial in the chain tension. (I've gotten pretty good at it!)
Ben