I admit that I do use mismatched panniers, and it doesn't bother me a bit.

I've often got some orangey-red Ortlieb rear panniers and a pair of Arkel front panniers which are a different red that almost clashes with the Ortliebs. I don't give it a second thought as both products are very well made and incredibly functional and that's my priority.