The brighter the better! The touring cyclists that I saw today ahead of me that had black panniers were barely visible. Who cares about a little dirt. As Bicyclebilly said, dirty panniers are a badge of experience. Erick is right that it's much easier to find things in a lighter colour pannier - it's amazing how things become invisible inside a black pannier (particularly inside a tent).