Yeah.

Also, if you have space at work where you can leave stuff (I work in a cube) you can dramatically cut down on the need to carry stuff. Shoes are easy to leave at work if you're riding clipless. Jeans aren't bad to. I leave a shirt because I work all day in a cube - it just doesn't need to be washed that often. Can be different for other people, but it's a lot less to carry just a shirt than it is to carry pants, shoes, and a shirt.

A pannier adds some drag to the ride. Not a ton, enough that I find it noticeable. I prefer a trunk bag which doesn't.