I used to use a Topeak rack trunk. Lunch, wallet, keys (if no pockets) went in the main compartment. Other things, including clothes got rolled and went into the fold down panniers.

Since I started grad school, I've been using the Axiom Lasalle panniers I've had for a year or two. This allows me to keep one pannier dedicated to change of clothes and lunch, while the other is school books, etc. That makes it easy to hop on the bike after stopping by home to head off for class with just one pannier, the school one. I generally roll clothes. Lunch goes on bottom (in tupperware), clothes on top of that and rest of lunch, usually bananas, goes on top of clothes to keep it from getting bruised too bad.