I am very on board with all of that, but that wasn't my problem when I did it. Here's what I mean:
I had a ~20L stuff sack strapped to a back rack. In it, I had my inflatable sleeping pad rolled in thirds, and that was the size of a can of beans. Around that, I packed the hammock and rain fly, and then I packed in my clothing around that stuff. After about 5 minutes, I managed to get everything packed tight, but 3/10 times I had to "redo" it because the stuff sack was lopsided.
Annoying.
Then, I used two straps to attach the stuff sack to my rack, and I'd also strap down a pair of shoes and a fleece sleeping bag under the same straps. Overall, I think I averaged 10 minutes. That doesn't include everything else, like getting dressed, eating something, untying hammock knots, etc.
In my current setup, I have a quick, loose stuff sack for things like the hammock, etc. I toss everything except my sleeping bag into one Ortleib and close it. I put my sleeping bag in the other Ortlieb and close that. So it takes a lot less time to take down camp. I still have a clothing bag strapped to a front rack, but since it's just clothes, I don't have to fiddle with it, and there's nothing else under the straps. I also cut my camp time by using a smaller inflatable pad, replacing hammock knots with carabiners, and simplifying the clothing worn.
Eh. Maybe I'm asking the wrong questions. There's something to be said for tossing everything in a pannier and getting back on the road, but even with a similar weight I notice a difference in handling between the "Ultralight" stuff sack setup and using two panniers.
Here's the bike: