I'm not sure how big a box you're putting on the rear rack. The bigger it is and the higher it sits the more it will affect your centre of gravity, and that will have an effect on the handling of the bike.

If you look, you can find wire baskets that attach to the rack similar to panniers. These give you the capacity without raising your centre of gravity.