Who cares what other people think? Wear what's comfortable.
Personally, I wear cycling clothes because my commute is long enough to justify them, and because I sweat enough to need wicking clothes in any weather. Otherwise I'd need a clothes dryer at work.
I do find this whole business of wearing team clothes wierd. I don't mean I think people who do it are weird, I just mean that I don't get it. For me, growing up, wearing team clothes was all about being a fan of a team -- baseball, football, etc. -- and you wore them when you were being a fan. But when you were actually playing the sport yourself, you wore your own team's uniform or, if it was a pickup game, you wore something unmarked.
I've carried that attitude with me into the present, and would feel uncomfortable actually cycling in some other team's jersey. So I wear plain cycling clothes.
I might wear a Postie jersey if I was watching a race, though.
RichC