I take pants, shirts, socks and underwear. I also carry lunch, basic toiletries and a small lock, as well as phone, keys and wallet. I leave my shoes, towel and D-lock at work, although I do change the towel occasionally. I use a backpack. It works well. It has two compartments. I put my clothes in one, and they don't really need ironing, and the rest of my stuff in the other.
I notice the difference when I'm riding without the backpack.