I just found a pair of "packable" shorts. They are thin nylon cargo shorts sold at Kohl's. They fold up into one of the pockets.

I also bought the large Cannondale seat bag (yeah, I had to get the Cdale one, I am such a poser) and it holds my spare jersey, shorts, spare tube, tools, with room to spare.