Mismatched panniers would bug me, but that's just me. (I'm also anal about my socks going with my jersey and my "kit" looking o.k. with the panniers.) My panniers have always matched because the two sets I have ever owned were bought as sets. If I used a handlebar bag I don't think it would bother me if it were a different color than my panniers as long as it wasn't ugly and did not clash with the panniers.