I see no downside from distributing weight more evenly. I always use four panniers and a handlebar bag.

One possible downside if you have your center of gravity too far forward, is that a really low gear can put a lot of torque on the rear wheel on a steep uphill. But you might not have enough weight on the wheel to keep it from losing traction. But this would be unusual unless you had some really nasty loose gravel conditions on the hill. I have had my rear wheel loose traction and slip a few times on such hills, and when that happens with a very slow speed with a really steep hill you can come to a stop extremely fast. But such conditions are uncommon.