Another vote for outdoor/technical clothing, though not necessarily bike specific.
I commute 3 miles each each way in normal work clothes, often with a merino T underneath, and Under Armor shorts instead of cotton.
The advice on layering is also very good. On cool mornings a light shell over my regular clothes works fine. I have a merino sweater/jacket that i can wear down to 40 or so, and putting the shell over that gets me down to 30 easily. At that point, ear, foot and hand warmth are a bigger issue than core warmth.
However, on weekend fun rides where I may ride 30 miles or so, I really like bike shorts for comfort, though I wear shirts that are a bit looser than a traditional jersey. More out of consideration for those who have to look at me than anything else!