I go shopping more often... it works out perfectly because I buy a lot of fresh fruit and vegetables that only last a couple days. For my biggest trips (if I run out of cereal, soy milk, oatmeal, pasta, beans, and all the other staples at once) I carry a backpack and use my panniers.
Good plan bringing your panniers in, I learned that the inconvenient way... the first time I used my panniers for groceries I overestimated how much could comfortably fit. I had to stop at the park on the way home and have a picnic, ate some of my fruit and snacks to free up some room. (best possible outcome, really)