This might be stupid, is your chainline straight?
Because you wear your crank arms everytime you install/remove them. This could mean that it might get drawn in deeper into the spindle each time, thus affecting the chainline.