No one answer to that one. I am fine with relatively tight quarters. My panniers stay loaded and on the bike. Only my handlebar bag (with my valuables and electronics), sleeping bag and pad, pillow, maybe a book, and clothes for the next day.
Some people feel the need to have more space and to take their panniers inside. You need to decide what works for you.