I think I have the same problem on a bike I brought back to life as a single speed. Riding straight, the top of the fork angles slightly to starboard. I was just going to replace it with a cheap eBay 1" threaded fork. What else besides a warped/misaligned fork would cause this?