I haul my work clothes and change as soon as I arrive. I leave my work shoes at work, seeing as I never wear them anywhere else.
I sweat too much in all weather for me to wear work clothes on the bike. Riding slower doesn't seem to make much of a difference for me, I'll still sweat. Even in sub zero F weather, I still sweat up my clothes even though at those temps I rarely break 10 mph. My hands and feet are like ice, but my torso is soaked.
Nobody has ever seemed to care if they see me come in or leave wearing non-work clothes. What matters is what I'm wearing once I'm at my desk or at the customer's location.