[From Kenny Kitzke (2007.07.19)]

<Bill Powers (2007.07.19.1310 MDT)>

<My opinion of statistical studies is still that all they give us is a lot of completely unexplained facts of very low quality. Of course that’s a statistical generalization.>

I feel your pain!

Before you give up on the value/merit of statistical studies, allow me to observe that I use statistical data and statistical analyses in my continuous performance improvement consulting with quite satisfying results.

I often see uses of statistical data in studies and experiments where conclusions are drawn and future results are claimed that are not justified by the analyses. Among the worst are using correlation as evidence of cause. Another is to apply population statistics as a reliable predictor of individual results.

Properly used, statistical analyses obey sound scientific/mathematical laws just like F=ma or a*2 + b*2 = c*2 for the hypotenuse of a right triangle. This can be valuable to understanding how systems work.

But, there are conditions for statistical laws to be valid. And, there are severe limitations on even using these valid laws to predict future outcomes. Let me address the first.

All statistical data is historic. Any predictive power is dependent upon the system that produced the data remaining functionally the same. This is true even for predicting future population outcomes much less trying to predict any particular individual outcome.

I often use dice rolling to illustrate statistical laws with my clients. Most have done this or you can do it live in front of them. They readily understand that the mathematical probability of possible outcomes predicts that the most likely next roll will be 7. They also readily agree that the next roll can not be predicted at all except that it will range from 2 to 12.

These statistical laws are so certain over time with large trials that casinos can count on making a profit if the system remains the same and is run the same way. However, if you change the system for “rolling” dice (so that outcomes are not equally likely), those laws will not work. For example, if the roller arranges the two die with sixes up between two fingers and drops them exactly one inch on to a cloth covered table, you will not necessarily find that 7 is the most frequent result in 1000 rolls.

Even if you can predict the probability of rolling a seven very accurately on the next roll, or of actually rolling more sevens than any other number in 1,000 rolls, the chances of rolling something other than a 7 on the next roll is greater than 80%.

Well, what may seem like bad news about the value of statistical analyses and predictions also has an upside. That is this. When a system is carefully characterized, you can intentionally perturb the system to see how the system change affects the resulting data. In other words, you can experiment to see how changes in a system actually change the results. It seems similar to a test for the controlled variable!

This reveals the only reliable way to improve any health care system. Looking at other systems will only give clues as to what might be a positive change for a desired new result. The same change in different systems can produce dramatically different results. So, if you want to see whether a single payer method is superior to multi-payer method in producing better health care outcomes, you have to take the current system and modify it ONLY in that way and record the results for a long enough time that you are convinced it is a helpful change.

I think you can see how difficult that is to do in a complex system with many variables that you can’t hold constant while experimenting one at a time. There are sophisticated statistical factorial experiments that would allow several levels of several variables to be evaluated for effect in one experiment. But, for very complex systems and ones that can change in ways that cannot be controlled, even these methods may not give precise answers. And, from my experience, very few organizations, even research organizations, are proficient at understanding and properly applying such advanced methods.

Lastly, humans are such complex systems. So the mysteries of human nature and behavior remain difficult to understand or predict even for populations and certainly for individuals.

See you soon at the University of Minnesota.