Did you decide?
I have Schwalbe Thunder \Burt (54mm) and Furious Fred (50mm). There is a HUGE difference, but that is buy design. The FF handles like a road bike tire and corners hard. The TB handles like a monster truck tire, and corners on pavement like a truck (although both do great on gravel).
Speed and ride quality based on size alone - no, not much difference.