The weight low down makes the bike turn sluggishly and there's always the issue of balancing the panniers. I like putting the weight under the saddle because for 90% of my riding I don't notice it. Out of the saddle it does make the bike feel a bit top heavy at first, but you quickly get use to it and I find myself climbing out of the saddle a lot. My handlebar bag has more of an effect on handling most of the time, but as you point out the low weight keeps that to a minimum.
BS alert.
If they are the same weight, the top-heavy bike will be much sluggish because you have to transfer your weight one side to the other. More you lean, it's tougher to switch direction due to the weight on top. You may "feel" it's okay, but it's simple physics.