Better to go with balance and symmetry. Grinding away mile after mile, thinking about the odd number of panniers, will eventually consume your every thought and drive you crazy. Do yourself a huge favor, and carry an even number of panniers, or single items loaded centerline only.