It's interesting you're considering going stoveless. I just did that on a solo X-C trip and it worked out great for me. Good luck and I hope it works out for you, too. That decision alone could get rid of one front pannier, at least it did for me.

On that trip, I used two rear panniers and nothing else, not even a handlebar bag, and just a foam sleeping pad on the top of the rack. I cycled through and camped in fresh snow in the Cascades, and I was comfortable in those conditions. One way to minimize your clothing bulk; you should be able to wear everything you pack at the same time. It's OK to bring two of everything, but they should work together as layers. For instance, I only carried one pair of shorts, but I also had a pair of light hiking trousers, some very light long underwear, and some rain pants. I could wear the hiking trousers with a long sleeve jersey at a nice restaurant or walking around town without looking like an alien. I could wear them all together on the wet, freezing 20-mile long descents in the Cascades. I could wear the rain pants on a laundry day. Or I could wash one item at a time in sinks along the way, and just "wear it dry" on a warm day.

If you do pack fewer than four panniers, you'll be in the minority out there.