Ah, so it's not so much as packing lighter, but more of how to carry it. When the OP uses the bike, s/he has to lug around a pair of shoes, helmet, and panniers after locking up. The option to drive in one day with a weeks worth of clothes and food is not viable at this point.

OP- you mention a "man purse" that you used when riding the bus. Is it something that is wearable like a backpack or messenger bag? If so, then maybe a separate bag to put your shoes in that you could carry by hand. Consider replacing the panniers with a crate or baskets. Below is a pic of what I use during the months were layering occurs.