I ride in regular clothes pretty often. I guess it depends what your 'regular clothes' are, tho. I usually don't put on long pants until mid-November, for example.

Many of my friends have trouble with this very issue, but this has more to do (in their case) with their clothes. I mean suits are fairly ridiculous for anything, though there are certainly exceptions.