Using stuff sacks will save a couple of pounds over more conventional bags, but at the cost of lots of straps and some inconvenience. Specialist bikepacking bags get around some of those issues, but people then proceed to stick bags all over their bikes which costs a lot and increases weight again. I find a single large saddlebag and a handlebar bag to be the most convenient for riding and getting around using buses and trains.
Even though I use them, bikepacking bags are definitely fussy, especially the handlebar rolls. There's something to be said for the convenience of a rack and pair of small panniers despite the weight penalty.
Nun, I'm surprised your legs don't rub against your Carradice. I tried it and it didn't work for me. Otherwise that setup of yours is nice and simple.