I find adjusting to the changes in my bike's steering from adding and removing front panniers (from and slow and steady to twitchy and responsive) to be FAR more irritating, distracting and potentially unsafe than the relatively minor changes in handling and weight distribution that result from adding and removing panniers on my rear rack. The only difficulty is occasionally losing my grip on the bike when trying to wrestle it indoors with one hand on the door and one on the stem. If I tilt it too far by mistake, the weight makes it very difficult to rebalance. Still, it's a lot easier than switching back to having no front panniers - the first few minutes are a bit frightening!

I suspect that most people who recommend front panniers alone have never used them. If I'm using one set of panniers, they go on the back. All of this goes double if I only need one pannier. One pannier on my front wheel? That would be insane and very, very stupid.