As you lower your bar, you change your reach, and your hip angle. So yes, it does mess with your fit, as your hip angle plays a reasonably big role in the fit. You'll also end up with slightly different weight distribution, etc.

Check with the guy you got fitted with.