Personally, if you aren't commuting every day, why deal with panniers? My commute route has a lot of hills so I try to minimize the amount of gear that I carry. Since I usually end up driving 1-2 days/week, I bring clean clothes, food and other gear in my truck and take home laundry. I use a Carradice Barley or Acorn Med-Large seatbag to carry my lunch and other gear on my bike, and usually have room to spare.

With a Bagman quick-release seat rack, you can quickly install or remove their Carradice bags. If you need to walk a few blocks, get a shoulder strap to attach to the rings on top.