Both work fine. Check out the ”Pictures of your loaded rigs” sticky thread.

Some find handling better with four panniers, weight balanced 40/60% and the lower centre of gravity.

Personally I find it all good. I only attach my front panniers for colder weather touring when I need the extra volume.

I leave my front rack on even when I don’t use it. Weight-weenies will break out in sweat at that idea