As with any front load that is attached to the steering components (handlebar, stem, forks), the heavier the load, the greater effect it will have on steering response. Even a load as little as an U-lock is noticeable. How much you can handle will depend on how the load is carried, and how comfortable you are with the results. Your best bet is to keep the weight as close to the steering axis as possible, where the weight will have the least angular momentum. Putting the weight lower, on a fork mounted rack instead of a handlebar mounted one, will also help.

Beyond that, you'll just have to try it out to see. It really bothers some people, and not so much for others.