Last week it was 104-105 when I got off work most of the days I rode. I'm not doing that in my office clothes. Even when the weather is nicer, I'm still going to be changing clothes when I get to work because I'm going to sweat, so I might as well wear something that's comfortable and dries fast for the ride home. Why not make that a jersey and bike shorts? From walking in the door, to sitting at my desk is usually less than 15min, and that includes putting on a pot of coffee.
Now, for pub crawls and such, its normal shorts and t shirts...
People generally wear bike clothes because they work well for riding a bike. Obviously, they arent always required, so do whatever fits the situation.