Bottom bracket drop is the geometry factor I’d be most concerned with in putting 650b wheels on the Diverge. Your Toughroad has a 70mm bottom bracket drop while the 3rd-gen Diverge is 80mm (the 2018-2020, 2nd-gen Diverge was a whopping 85mm). With 650b wheels, the decrease in ride-height creates that much more of a pedal-strike issue if you’ll be riding off-road around rocks and tree roots. I have a 2020 model and decided to skip on 650b for just that reason. You may be aware of that and perhaps is why you specified handling in your question.