When I take my panniers, I always take two as well. Guess it's the symmetry thing. In my case, too, if I'm taking the panniers on my commute, chances are really good that I still have my rack trunk on the rear, so the panniers go on the front. Not only would I be constantly reminded of the fact I only have one pannier on the rack (since it'd be in plain view), but the weight distribution seems to matter more on the front. All the weight to one side and it makes the bike want to turn that way. So, OCD or not, I'm with the OP.