For conversions, probably because or brake reach. Fitting 650b wheels on a frame that was designed for 700c or 27" rims can be doable with longer reach brakes. 26" is just a bit too far to make such conversions possible.

I'd expect 700D to be a harder sell for exactly that reason -- decades of history and tradition and accomplishment in favor of 650B, a little-remembered flop in the case of 700D. (No offense, but can you name anything else that wore those tires besides GT mountain bikes for a couple of years? ;))

I think "why 650B over 650A?" is a much better question, which some Limey alluded to a page or so ago. 650A is also in the ballpark of these other standards, almost exactly in between 26" and 700C rims in diameter, and was a pretty well-established standard for many years. Nice tires and rims in that size would be a great addition to an existing (if not currently sexy) market.

Cachet, probably. Everyone knew that size as the "tiny women and junior high schooler" size.

People (both historically and recently) seem to want a size that's in-between 622 and 559. As others have pointed out, that way you can have three different tire widths but a similar overall diameter (which the French decided should be ~670 mm).
[MENTION=339610]Darth Lefty[/MENTION], why 650b/584 specifically over 587 (or 590)? Because Japanese bike enthusiasts propped up the vintage French Randonneur bike market throughout the 90s. They went gaga over the retro style, and liked the riding characteristics (I guess). So, they joined in with the French retro-randonneurs (and their Confreries de 650b) to make 650b/584 rims and tires. (most modern rims and tires are made in the far east, after all).

The French started it, the Japanese kept it going, and then Americans jumped on it.

Where it got weird: the American 650b resurgence of the 2000s was happening at the same time as the "29er" mountain bike wheel.
Soon, folks realized that the 29er wheel is very tall (busting the old French ~670 mm overall diameter paradigm), which gives it that excellent steamroller effect you describe, but it's too tall for shorter people.
Kirk Pacenti (and Panaracer) started making the 584 Quasi-Moto tire, and "27.5" mountain bike wheels were born. They're closer to the French paradigm, so they fit better for a wider group of riders.
New, 27.5" and 29" MTBs are popular and 26" ones are very rare.
Disc brakes have freed us from fixed wheel sizes on a frame, so folks are further encouraged to experiment (see fat bikes, 27.5+, and 29+) to find wheels with specific characteristics for specific riding.

Personally, I'd like to see more 650b/27.5 all-road/'cross/gravel bikes, for the shorter rider that still wants to fit medium-width tires. 622 x 40 wheels are really hard to design around for frames smaller than 50-52 cm. The Surly Straggler is a great example (not without its problems, ahem dropouts?) with smaller frame sizes available in the 650b version.
Thankfully, with disc brakes, it's easy to swap 650b/584/27.5 wheels into a 'cross/touring frame.

My girlfriend wants a light 'cross/gravel bike (like my Foundry), but she's 5'3". I'm suggesting that she gets a disc 'cross frame and fit it with 27.5 wheels. She works at a Trek/QBP shop, so I suggested the matte purple Crockett disc frameset or the Foundry Camrock frameset. She already has a steel 650b road/touring bike (Soma San Marcos), so I don't think she'd find the Straggler different enough (not much lighter).

here's an interesting article:

Converting From 26inch to 650B on a Touring Bike - CyclingAbout CyclingAbout

It depends entirely on the frame, not necessarily the factory-designated wheel size.

In a another thread, I mentioned that my first 650b conversion was an 18" Trek 820, and I converted this bike as an experiment (which ultimately worked out quite well). There was only one drawback: 650b rims were indeed too wide for a standard canti on the rear wheel; the front fit fine, but the rear was too much of a reach. I got around this by fitting the bike with a set of Tektro 'Bigmouth' 556 calipers and leaving the canti bosses bare. If I'd felt like swapping out my brake/shift levers for V-Brakes, I could have simply fitted those without issue. As it was though, the bike rode extremely well, and my braking power was greatly improved. I rode it like that for about two years, until I found a larger 20" 820 frame and transferred the components to it, albeit with a different set of brakes (Avid Shorty 6); it came together neatly and it works fine. I still have it, and it was my daily ride for four years. FWIW, You can also do this with a Surly LHT, which is very, very similar to the 820 (even though it's almost a clone of the '84/'85 Trek 850/870).

I sold the first 820, with it's original brakes & wheels restored, and, after building my 650b Panasonic (which apparently is, judging by its odd geometry, canti boss placement, & how well & exactly it all fit together, an actual 650b-native frameset), I realized I had enough spare parts for a fourth 650b build, albeit a very unique one. So I went on the hunt for a very unusual type of early '80s MTB frame, one designed for caliper brakes. There were several of these; the '85 Bridgestone MB-3; the '84 Panasonic Villager DX; the '85 Nishiki CitySport; the Raleigh Yukon/Grand Mesa (which, as far back as 1985, was specifically designed as a 650b frame)... and the one I found, a Univega Rover 10, which made a perfect 650b city bike (and cleared out a lot of spare parts- including the Tektro brakes used on the first Trek conversion).

All of these bikes rode/ ride exceptionally well and were a lot of fun. My current 650b project, restoring a Toei Randonneur, will likely be the best yet.

And let us not forget the Gospel according to Brooks, and its prophet, St. Proofide.

Toei
 

[MENTION=304366]DIMcyclist[/MENTION]

Please make a thread about your Toei!

Not available yet, but soon: Pacenti Forza and Brevet.