If I only have a mile or two to travel, I can wear anything. But if I'm traveling farther, by about a couple of miles my body temperature starts to climb and that's when it's really nice to be wearing something that can wick and breathe. For this purpose it doesn't have to be biking specific. Inexpensive sportswear can be found at just about any department/big box store. In the heat of summer I often wear sportswear I bought at Target.

For the rain I have Pearl Izumi rain gear. It's breathable (sort of) and serves a decent wind barrier when the temperature drops.

I do prefer the bright colors of cycling clothes. I really couldn't care less what people think of it when they see it, what matters to me is that they CAN see it.