I've never seen a brake rub hard enough to cause a crash, though it's possible with double pivot brakes which have less horizontal float. OTOH, if a wheel is untrue enough the tire can rub the fork blade hard enough to stop the bike and cause an endo.

This risk, and that of squirrels or road debris getting jammed between the spokes of low spoke wheels is why I won't use them for front wheels. These kinds of things are rare, so they don't rank high on the table of risks, but since I don't ride competitively I see no need to take them on no matter how small.