Leave a pair of shoes at work, they always take up too much space. Maybe you can get a small frame pack like what you would use for bikepacking. Something similar to the Specialized Burra Burra frame pack (and there are many other brands) should hold a change of clothes without adding to your wind resistance.