You mean the symmetry of the spindle will be off? Because from the factory it's not symmetrical unless my cranks or pedals compensate for it (haven't measured). Adding the spacer would actually make it closer to perfect symmetry, albeit a little in the other direction.

And I don't know if the symmetry would affect the pedals as much.. what is your reasoning for this?