Statistical Correlations and Conclusions

[From Rick Marken (2003.03.06.0900)]

Dan Miller (2003.03.06.10:40 EST)

What Richard Kennaway and other statisticians do not understand is that
statistical correlations between
two or more variables is not the endpoint of inquiry. Rather, such
correlations indicate that the relationship
is not random.

Actually, such correlations indicate that the relationship _may_ not be (or have
been) random (a chance association). And even if the correlation is almost
certainly not random (a correlation of .99 for example) that does not mean that
the variables are actually functionally related (see below).

A pattern exists - however paltry. Any good social or
behavioral scientist would take this
heuristic evidence and continue the investigation with a next stage in
which the investigator designs a
study that will produce more accurate and definitive evidence.

To the extent that this ever happens it just involves holding some variables
constant or measuring concurrent variations in other variables that are not
involved in the correlation and seeing whether, by statistically removing the
variance in these variables, the original correlation can be improved. Sometimes
there is some improvement in the correlation -- going from .3 to .7 would
considered a huge improvement -- but you don't reliably see the kind of
correlations you get in PCT research (.99), where we actually understand what is
going on in terms of a well-tested working model.

At some point in the process of inquiry
a pragmatic test should be conducted. I believe that Perception Control
Theorists understand this stage
of inquiry.

If "pragmatic test" means "comparison of actual behavior to the behavior of a
model" then I would say that this is the way the entire process of inquiry is
carried out in PCT. Relationships between variables mean nothing unless they are
understood in terms of their relationship to the behavior of a working model.
Indeed, it's easy to show that there can be very high correlations between
variables when there is no functional relationship between these variables. For
example, look at the correlation between disturbance and handle in a compensatory
tracking task, which can be on the order of .99. This correlation is certainly
non-random. But it does not reflect a functional relationship between disturbance
and handle. The disturbance variable is not even visible to the subject in a
tracking task.

A control model shows that the high correlation between disturbance and handle is
actually a side effect of the control process, in which error driven handle
movements compensate for the simultaneous effects of the disturbance on a
controlled variable. Without the model, the correlation between disturbance and
action is likely to be interpreted as a causal relationship, with disturbance
being the cause of action (handle movement). And, indeed, this is exactly what
happened in the behavioral sciences. Correlations between disturbances and actions
are still seen (incorrectly) as causal relationships. If the apparent causal
relationship between disturbance and action had been investigated in terms of a
model (as was described in the "Models and their Worlds" paper by Bourbon and
Powers) the correlation between these variables would have been seen for what it
is: the behavioral illusion.

Best regards

Rick

···

---
Richard S. Marken, Ph.D.
Senior Behavioral Scientist
The RAND Corporation
PO Box 2138
1700 Main Street
Santa Monica, CA 90407-2138
Tel: 310-393-0411 x7971
Fax: 310-451-7018
E-mail: rmarken@rand.org

[From Dan Miller (2003.03.06.10:40 EST)]

Bill Powers (2003.03.06.0737 MST)

Gary Cziko reported, or found a report that said, that the average
correlation in a large sample of published social-science studies was 0.26.
This led to a discussion on CSGnet about the kinds of correlations needed
to draw justifiable conclusions; Richard Kennaway came up with a full-scale
analysis showing that most conclusions drawn from studies even in the
higher range of correlations usually found are unjustified. That, in turn,
gave rise to a number of highly defensive "yes but" comments, and the
subject was eventually dropped. The conclusion that statistical "facts" are
next to useless for screening or reaching other conclusions was evidently
not enough to raise the level of skepticism about "knowledge" based on such
"facts."
Dan Miller writes,

What Richard Kennaway and other statisticians do not understand is that
statistical correlations between
two or more variables is not the endpoint of inquiry. Rather, such
correlations indicate that the relationship
is not random. A pattern exists - however paltry. Any good social or
behavioral scientist would take this
heuristic evidence and continue the investigation with a next stage in
which the investigator designs a
study that will produce more accurate and definitive evidence. At some
point in the process of inquiry
a pragmatic test should be conducted. I believe that Perception Control
Theorists understand this stage
of inquiry.

Dan Miller
Department of Sociology
University of Dayton
Dayton, OH 45469

[From Bill Powers (2003.03.06.0901 MST)]

Dan Miller (2003.03.06.10:40 EST) --

>What Richard Kennaway and other statisticians do not understand is that
statistical >correlations between two or more variables is not the endpoint
of inquiry. Rather, >such correlations indicate that the relationship is
not random. A pattern exists - >however paltry. Any good social or
behavioral scientist would take this heuristic >evidence and continue the
investigation with a next stage in which the investigator >designs a study
that will produce more accurate and definitive evidence. At some
>point in the process of inquiry a pragmatic test should be conducted. I
believe that >Perception Control Theorists understand this stage of inquiry

I suspect you're speaking of the Ideal Behavioral Scientist rather than
those that actually publish. Most studies that are published are never even
cited by anyone, much less replicated or followed up with refinements for
more than a year or two. Even when a work is cited, it is normally the
original work that is mentioned and not any further developments (or
refutations). Look at all the references to left-brain-right-brain stuff
since that idea was first brought up. Tom Bourbon, who has done such work,
advised us that the differences in activity that are found amount to about
5% of the activity on the two sides, in the intact person. A so-called
left-brain activity is accomplanied, in other words, by about 95% as much
activity on the right side. The same is true of most studies. Mothers hold
their babies on the left, have you heard that one? There are literally
hundreds of other facts like this, probably thousands. Perhaps you are a
shining exception to the rule; I hope so.

Come to this years meeting and argue about it!

Best,

Bill P.

[From Dan Miller (2003.03.06.11:35 EST)]

Bill Powers (2003.03.06.0901 MST)

I suspect you're speaking of the Ideal Behavioral Scientist rather than
those that actually publish. Most studies that are published are never even
cited by anyone, much less replicated or followed up with refinements for
more than a year or two. Even when a work is cited, it is normally the
original work that is mentioned and not any further developments (or
refutations). Look at all the references to left-brain-right-brain stuff
since that idea was first brought up. Tom Bourbon, who has done such work,
advised us that the differences in activity that are found amount to about
5% of the activity on the two sides, in the intact person. A so-called
left-brain activity is accomplanied, in other words, by about 95% as much
activity on the right side. The same is true of most studies. Mothers hold
their babies on the left, have you heard that one? There are literally
hundreds of other facts like this, probably thousands. Perhaps you are a
shining exception to the rule; I hope so.

Dan responds:

Bill, you are absolutely correct, I am talking about an Ideal Behavioral
Scientist. I've been tilting at this windmill for the past twenty years
or so. My feeling is that most research that is published is done so for
administrative reasons, i.e., to get tenure, promotion, or a merit pay
increase. Yes, cynical, I know. A few scientists out there understand
the limited heuristic value of correlational studies. And, as you note,
many of those studies have no value whatsoever. I try to be an exception
to the rule, but, then, I've had good teachers.

Bill suggests:

Come to this years meeting and argue about it!

Dan replies:

I'll try. I've been away too long.

Always,
Dan

Dan Miller
Department of Sociology
University of Dayton
Dayton, OH 45469

[From Richard Kennaway (2003.03.06.16:06 GMT)]

Dan Miller (2003.03.06.10:40 EST):

What Richard Kennaway and other statisticians do not understand is that
statistical correlations between
two or more variables is not the endpoint of inquiry. Rather, such
correlations indicate that the relationship
is not random. A pattern exists - however paltry. Any good social or
behavioral scientist would take this
heuristic evidence and continue the investigation with a next stage in
which the investigator designs a
study that will produce more accurate and definitive evidence.

Can you give some examples from the literature of this process in action?
Even if a followup study produces a correlation of 0.6 (unusually high for
a paper in psychology, so I understand) that's still far too mushy to use
as a basis for individual predictions.

And, that's leaving aside the question of whether studying
stimulus-response correlations of any size at all is the right thing to do.

-- Richard Kennaway, jrk@sys.uea.ac.uk, http://www.sys.uea.ac.uk/~jrk/
School of Information Systems, Univ. of East Anglia, Norwich, U.K.

[From Bill Powers (2003.03.06.1109 MST)]

Dan Miller (2003.03.06.11:35 EST)--

Bill, you are absolutely correct, I am talking about an Ideal Behavioral

Scientist.

That's a relief. I knew you were one of the Good Guys.

>>Come to this years meeting and argue about it!

Dan replies:

I'll try. I've been away too long.

It will be a pleasure to see you there. You always have something
interesting to say.

Best,

Bill P.

[From Kenny Kitzke (2003.03.06.13:05 EST)]

<Dan Miller (2003.03.06.11:35 EST)>

<A few scientists out there understand

the limited heuristic value of correlational studies. And, as you note,

many of those studies have no value whatsoever. I try to be an exception

to the rule, but, then, I’ve had good teachers.>

After reading that a scientific study of 1,000 suicides showed that 95% of them grew up as children in bedrooms painted white or off-white, I made sure dark colors of paint or wall paper were used in the childhood bedrooms of my three sons.

Sure enough, all three of my sons are still alive! Where would the world be without psychologists?

Kenny

Certified in Statistical Shewhart Control Chart Methods

[From Dan Miller (03.03.06.15:20 EST)]

Kenny Kitzke asks where the world would be without psychologists.

Dan Miller replies:

Right where it is, now.

Cheers,
Dan Miller
Department of Sociology
University of Dayton
Dayton, Ohio 45469-1442
937.229.2138
Dan.Miller@notes.udayton.edu

[From Bruce Gregory (2003.0306.1646)]

Dan Miller (03.03.06.15:20 EST)

Kenny Kitzke asks where the world would be without psychologists.

Dan Miller replies:

Right where it is, now.

Are you sure we'd be no better off?

···

--
Bruce Gregory lives with the poet and painter Gray Jacobik in the future
Canadian Province of New England.

www.joincanadanow.org