My main concerns with the duffel would be a) the duffel not being secured completely to the rack in all dimensions (so that you have a loose end flopping off the back) and b) the duffel shifting to one side or another and unbalancing your load that way. It also would be inconvenient to access stuff in the main compartment of the duffel while you're on route, so that would influence how you pack the duffel.

I don't think that the weight itself would be so much of an issue, provided that it is secured completely and doesn't shift. Getting a load that is secured completely and doesn't shift might be a tall order, though.