I commute with a pannier that is mostly empty most of the time, because I never know when I'm going to pick up a gallon of milk on the way home, or wind up taking something home from work. Also, it's more convenient than my trunk bag, it goes on and off faster. The bigger bag is nice because I like to take a week's worth of clothes in at a time rather than one day's at a time, because it gives me more flexibility to not try to bring clothes in during a rain storm where they might get wet/dirty, and have some spare clothes in case I spill something on myself during the day, etc.

Also, sometimes I find cool stuff on the way home, and having a big bag and some bungie cords in it helps me get stuff home sometimes.

I don't think having a big bag with me that's mostly empty hurts my commute in any way, so I don't feel much need to go out and buy a smaller bag which I would have no reason to have.