It may or may not be an issue. If the head tube ends are not parallel to each other, (and perpendicular to the head tube), and the steerer is not centered, it will probably cause more stress on the bearings at a particular point since any impact will not be absorbed evenly over the headset bearings. How this effects longevity, in miles, is hard to say.

It may also impact how smooth the handlebars turn through the entire arc.

If it is a spacer issue and you are able to pre-load the bearings so they are smooth with no play, it probably has no impact once the stem is tightened to the steerer tube.

John