I have two large panniers and recently added a crate to the top of my rack and I don't really think that would be enough for groceries for a family of four and I worry about the rack, plus it makes for uncomfortable ride when loaded. I have a trailer too and it makes hauling significant weight much easier: no tail-wobbling and it behaves better in crosswinds. I totally second the idea of a trailer. Burley has an attachment that goes on the chainstay and I bet others do too. DIY trailer might be cheap but if you buy a ready trailer you may find one that folds flat for storage which is a plus.

BTW, Yuba Mundo is the Master Hauler and it stands up on its tail for storage, so it takes up less floor space than a regular bike.