I've cycled in all sorts of things - three piece suit to work, even morning coat when going to a wedding. Neither was a long journey nor regular and - obviously - looking peculiar is not a big worry (while cycling or otherwise). Only once has my boss asked me to do him a favour and never wear the shirt I was wearing again. Even so, I do sometimes wonder whether the wisdom "other people's opinion of you is their problem" isn't one I over-rely on.
For myself, I prefer dark "ordinary clothes" - dark because it hides the stains and ordinary clothes because people who aren't cyclists seem to react better to them. Like others, I get the theory of panniers but still find a messenger bag is what actually works.
I also seem to curse clothes by liking them - Levis commuters and Wolsey grip top socks are pretty much my standard wear.