I got into the business of testing and its applications about
1957. Even by that time, the impossibility of doing any good for one
particular person without a validity correlation in the high .90s was
widely known among commercial users of tests such as Educational Testing
Service, clinical psychologists, forensic psychologists, etc etc. I have
always been baffled by the fact that all those people have gone right on
passing out their tests to schools, prisons, etc, where they know they
will be used to make decisions about individuals.
I have been equally baffled by the thousands of social researchers
who claim they are digging down to the truth about human functioning by
pretending that every individual is the same creature as the average
creature calculated from their latest sample of subjects.
But I understand some of that, because I once suffered from both
those blindesses -- or myopias, or compartmentalizations, or
afflictions, or stupidities. I still have to watch myself. And don't
forget that the kind of statistics we are complaining about are still very
useful when all you want to do is cast a net -- when you want to catch a
large proportion of something and don't care what individuals show up.
Welcome to Kennaway.
[From Bill Powers (970426.2329 MST)]
From: David Goldstein
Subject: Correlations in individual case
Date: 04/27/97
This statement by Phil goes along nicely with what I posted before and
will one more time. If a person has taken Test X, and the person's
location in the distribution of Test X scores is used to predict the
person's location in the distribution of Test Y, and the standard is
"must be in the same tenth as predictor test" , here are the chances in
100 that this will occur:
Correlation between Test X and Test Y Probability
.90 55/100
.95 71/100
.98 91/100
1.00 100/100
By chance, if the test Y distribution is divided into ten parts, 10/100
times will one pick the correct answer. With knowledge of Test X, the
probablity increases.
If this were a decision of great importance to me, and I had to go
along with it, the smallest acceptable probabiltiy would be .98, which
means that 9/100 times it would be wrong for me and 91/100 it would be
right for me. This requires a correlation of .98.
If we insist on this, we are going to save a lot of trees.
Bravo, and hear, hear! I presume everyone has heard of the Worm Runner's
Digest, also called the Journal of Irreproducible Results. Speaking of
saving trees, I propose that somebody start a psychological journal called
"The journal of accurately reproducible results." It would accept only
reports in which correlations are 0.98 or higher. I believe it could be
published quarterly, and could be edited, proofread, typeset, printed, and
distributed by a staff of two people.
Best,
Bill P.