"does the marginal size difference between 584 and 622 make a significant difference?" Yes it does, and it's not that marginal.

I've heard that for smaller riders 700C can feel a bit too big, but I can't comment on that since i'm 6 feet.

The true benefits of 650B come when you try to jam a big tyre in a drop bar bike. MTBs use a wider bottom bracket, they have longer top tubes, and slacker head angles, long forks with small headtubes. Gravel or adventure bikes are just tweaked road bikes.

Grab a road bike frame, tweak the geometry slightly, then you get a gravel/adventure frame. Now try to design it to accept a 700x47-50mm tyre. It's not going to be easy. At the back, you have the 68mm bottom bracket WITH bigger chainrings, that limits you a lot. So unless you do something like a chainstay yoke, or an asymmetrical dropped chainstay you simply have to lengthen the chainstay a lot. Then you get to the front. You're going to get a LOT of toe overlap with a big 700C tyre and something like a 71-72 degree head angle, and shorter top tubes compared to MTBs.

650B makes sense because of the smaller diameter you can easily retain the road-ish geometry with big tyres.