Kinda depends on the size. I've seen some trunks with the drop down panniers that were equal or greater than some of the smaller panniers.

@OP- Other than the clothes and shoes, what else do plan on carrying? Rain gear? What about lunch? There's also another option you could explore, since you don't have a rack yet- Carradice.