If you have a spot for it, leaving clothing at work is really the way to go. I wouldn't want to walk through work in spandex, but I just wear mountain biking shorts and they work great. A pannier adds air resistance - for me at least, the kind I can feel rather than the kind that's theoretically making me slower. It makes the bike handling slightly more annoying because it's heavier and weighted on one side now. I vastly prefer a trunk bag.

If you want a pannier, you can get a pannier/backpack combo like the arkel bug:
https://www.youtube.com/watch?v=zYEmp4toAck

That's a pannier on your bike, and a backpack off of it.

If you want a trunk bag, several different companies make bags and quick release systems. Topeak makes the most commonly found - bag comes off by pushing down on a button and pulling it off the rack. It's very slightly noisy for me though, where the plastic attaching pieces goes through the metal guide. Bontrager (and others) also make more recent versions that attach without the metal guide that I've been thinking of getting myself. Bontrager even makes a rack for them that goes on full carbon bikes without rack mounts.