My experience is that you need to add some non-zero share of your load to the front. It doesn't matter if it's front panniers or handlebar bag or a front rack-top bag or load. Just not 100% on the rear as this affects handling negatively, especially at slow speeds or when trying to push the bike beside you while walking - a bike heavily loaded only over the rear axle may want to fall or flip over suddenly.
So if you already have a handlebar bag, carry some of your heavier but smaller items in it to balance the bike better. It doesn't need to be much, just a fraction of whatever is carried on the rear.