The single point attachment at the seat stays or brake bridge does not have to be particularly strong. It's only there for for/aft stability.

The most stable racks have some triangulation when viewed from the rear. A rack with all parallel vertical members will be much more prone to swaying. Larger diameter tubes help, but that's a "band-aid" for lateral stiffness. If you look at the last vertical member in that photo of the old Blackburn rack, note that it is narrower than the two other members. They're the ones carrying the pannier load, but that last one is what keeps it from swaying.