Rear trunk rack plus pannier bags.

It's the solution that was specifically designed to address the problem you want to solve.

The bags hold securely to the rack when riding but pop off with ease (no bolts, snaps, fasteners, etc) when you don't want them.

For carrying a full set of clothes in a single pannier I recommend a minimum of a 15 liter bag. And that's it's you have small feet (and thus small shoes). If you've got size 11's or bigger make the minimum an 18 liter bag.

A pair of 15 liter bags (one on each side of the bike) will get most anyone through their day with a full change of clothes, lunch, a book, maybe a laptop, etc.