If I am going to change anyway (particularly commuting) I wear road bike shorts and a jersey. If I am riding more than 10 or 15 miles I wear bike shorts. If I am tooling around town I just wear street clothes.
Cycling clothes make a whole lot of sense -- they are comfortable and help you ride better. They deal with all the sweat and keep you cool. I love modern fiber technology, even though I ride a 30-year-old bike. I have a real weakness, however, for vintage cycling clothes, especially wool jerseys.