Another even smaller effect is the small decrease in geometric trail due to the smaller wheel. But we are talking about a 1-2mm difference (calculator here) so it's not noticeable.

In terms of handling well bigger tires are more "grabby" when turning. In terms of rolling resistance I don't think there is a big difference. They are definitely more comfortable. Note that I haven't tried 650b specifically but 700x35c and balloon 26x2".