It just looks imbalanced - when I'm not planning on a large load I only take one pannier. Just to carry my extra tools/tubes etc.

I can imagine it would cause a problem if it was overloaded and heavy - but if thats the case just take them both!