makeinu

I had started preparing this little thread for the A&S forum a while ago, but given my recent accident I don't think I'll have time to finish it. Rather than letting it go to waste I figured I'd just put it out there and see what kind of feedback I'd get:

In the thread, Evaluating advocacy and safety arguments in pursuit of knowledge, invisiblehand sparked a discussion (based on this article) on the role of heuristic reasoning when applied to the general problem of understanding the consequences of cycling behavior and advocacy in the real world. Among other things, we discussed a little of the theory behind statistical reasoning and compared its potential to that of other heuristics. In order to flesh out this discussion and make it more concrete I'd now like to try to reproduce it in the context of a real world example.

In the thread, Just how dangerous IS cycling?, Bicure sought a statistical answer to the question posed in the threads title. Here was his original post:
These question(s) implicitly reflect the primary heuristic of the traditional statistical framework which is that the truth about an event of interest (in this case the early death of the individual who wishes to avoid death) is best understood and predicted as a random event. From the Frequentist point of view this means we consider the degree of veracity, or truth, about the occurrence of an event of interest to be proportional to the frequency with which the event would occur relative to the number of times the circumstances surrounding the event could, in theory, be exactly repeated. In other words, the heuristic used to determine the truth is as follows: if the relative frequency is high then our confidence in the veracity is high and if the relative frequency is low then our confidence in the veracity is low. This fundamental heuristic is common to all of traditional statistical theory and, while they may not admit it, the primary difference between this point of view and the Bayesian approach is in the burden of proof. That is, although the heuristic is the same, the Frequentist approach requires that all statements of truth be accompanied by empirical measurements of relative frequency (or mathematical extensions of such), while the Bayesian approach allows for the ad hoc assumption of relative frequencies. Nevertheless, it is within this common heuristic that Bicure's formulates his questions about the danger of cycling.

In accordance with a technique most typical of statistical analysis Bicure constructs the following strategy: As an individual he wants to avoid an early death. So he considers two different sets of circumstances and tries to use the relative frequency heuristic to measure the veracity of the event in each scenario. From this he concludes that if he effects the circumstances for which early death has a lower veracity then he has, in fact, made the event of early death less true for him.

Unfortunately, statistical theory has not given Bicure a means to identify the circumstances which he should be considering.
The nature and complexity of the true circumstances are not admitted by those selected for the analysis. In particular, there is the extremely confounding phenomenon of feedback. People are not like playing cards. Effecting the circumstances also changes the circumstances because people immitate each other, influence each other economically, etc, etc. In some ways driving encourages others to drive and in others it discourages others to drive and likewise for cycling. Moreover, many telling circumstances such as rider skill, riding purpose, etc, etc are neglected due to insufficient data. In the end this all contributes to a statistical method which falls quite short of the full potential of the complete statistical theory.

It turns out that these kinds of problems are not at all uncommon when it comes to applying statistical theory to real world questions and have spurred many to seek out alternative methods. Although there are many alternative heuristics available, a typical complaint is that, when applied, they lack the precision, the proclivity for strong argument, and the reproducability of the traditional statistical theory. Fortunately, there is at least one alternative heuristic measure of truth which overcomes these drawbacks: falsifiability.

The heuristic measure of truth known as falsifiability says that the veracity of rule used to predict events is best measured by the number of observations which are not counterexamples relative to the ability of the rule to, in theory, encompass all possible examples. Note that the relative frequency heuristic measures particular events from which rules can be derived, whereas the falsifiability heuristic measures rules from which particular events can be derived. A common demonstration of this heuristic is Occam's Razor, which is often quoted as "the simplest explanation is the best". Since simpler rules should, in theory, be more easily proven false by counterexamples, the falsifiability heuristic says that the failure of such counterexamples to surface in observation indicates that there phenomena will probably never generate many counterexamples (ie rule must be fairly true).

outline for additional text:
1. Deterministic vs statistical (deterministic is more falsifiable, indeed unlike a statistical model a deterministic rule can be falsified by even a single counter example).
2. Select the simplest and most deterministic rule consistent with available data as a predictor.
3. Explanation of varied/anecdotal observations using all variables (IID sampling only exists under carefully prepared circumstances. Instead of trying to filter-out/prevent nonidentical circumstances in real world data to justify IID arguments, use the unique circumstances surrounding each data point as high dimensional inputs to a general deterministic rule...only a statistician or a lunatic expects different results from repeating the same experiment ).
4. algorithms (Support Vector Machines, etc)
5. computation vs data collection (Heterogenous/varied/anecdotal examples require less effort in data collection but massive amounts of computation to construct the simplest consistent model which accounts for all variables. Traditional statistical methods require painstaking highly controlled data collection but little computation).
6. potential attained vs failue to attain potential (Since varied/anecdotal examples and computational power are easily available, while controlled sampling and clever statistical models are not, predictors depending on falsifiability heuristics may come closer to attaining their full theoretical potential).
7. human reasoning (more heuristics?)

Metricoclock

^that was a good read and commentary on cycling and safety