Is there any slop (looseness) in the headset? Stand next to the bike and hold the front brake tightly. Rock the bike forward and backwards while checking for movement between the fork crown and the head tube. Should not be any looseness.

If either side of the rear axle is moving in the dropout it can cause the bike to wander. This is a common problem with horizontal dropouts. I'm thinking you should check what's going on at the back wheel.