I tried riding to work in my normal clothes for about a week. That was 2 years ago. Now, it's all bike shorts and a florescent colored jersey. I just recently started doing the same thing for church. The associate pastor did a double-take when some sweaty twenty-something in spandex came through the main doors and headed straight for the men's room.
The only time I ride anywhere in socially acceptable clothing is to the Dr's office or to Wal-Mart or something. Even then, I'll just wear a light shirt and light shorts (like swimming trunks) over my bike gear.