[From Bill Powers (2010.03.16.1915 MST)]
Bruce Gregory (2010.03.16.2041 DST) –
BP earlier: I have objections to
conventional methods that would hold even if PCT didn’t exist. The use of
population statistics to explain or predict individual behavior is, to my
mind, a mistake that needs to be rejected whenever it’s made.
BG: Would you propose that we outlaw insurance that is based on
population statistics?
BP: No. See Phil Runkel’s Casting Nets. Population statistics are
wonderful for predicting the behavior of populations. However, the
relationships among variables that hold for a population are quite
different from, and can be completely opposite to, the relationships
between those same variables in an individual. You may have missed
Richard Kennaway’s proof of that general principle.
Population statistics work to the advantage of people and organizations
who deal specifically with populations and have no interest in what
happens to any specific individual. An insurance company can predict the
accident rate among teen-age drivers and adjust insurance rates
accordingly. But it can’t predict which teen-aged driver will have an
accident – nor, if insurance companies played by the rules implied in
the concept of insurance, would they try to. They wouldn’t even ask the
ages of drivers.
BG: I can imagine life insurance
based on a detailed analysis of DNA and life style. Some people would pay
less and others a great deal more. Would that be an improvement over the
present system? How would a company decide what to charge me for
insurance on my house? For flood insurance?
BP: I think I’ve taken care of that. Insurance is supposed to spread the
risk according to population statistics, so no individual will be
devastated by rare high-expense problems. But insurance companies love to
do more detailed studies, so, for example, they can collect premiums for
health insurance only from healthy people. I hope that becomes
illegal.
BG: Should I ignore the
statistics that say flying is much safer than driving? How should I make
the decision whether to drive or fly? Some people live to a ripe old age
while smoking two packs a day. How do I know whether I am one of them?
Should I smoke to find out? How about taking Vitamin D? How do I know it
will do me any good? Should I take more Vitamin D or less? On what basis
should I decide?
BP: The only thing you have to go on is population statistics in cases
like these, though you can do some cherry-picking by selecting behaviors
that generally have better outcomes. Some airlines have better records
than others. However, the fact that it is vitally important to get the
predictions right in these cases does not improve your ability to predict
correctly. I remember Bill Williams saying that since financial
institutions handle so many billions of dollars every day, and the impact
on human lives is so enormous as a result, all the criticisms of economic
theories must therefore be wrong. After all, people carrying such a great
responsibility wouldn’t use a wrong theory, would they? It was in vain
that I protested that the importance of a prediction has no effect on its
correctness.
BP earlier: Interpretations that
go beyond the evidence and assumptions that are untested should be
considered as hypotheses to be tested or dismissed as unjustified (like
the assumption that placing a mirror in a room with someone increases
self-awareness).
BG: The studies of which I am aware make no such assumption. They look at
behavior with and without mirrors. I don’t see what is wrong with that.
What am I missing?
BP: How do they verify that putting a mirror in a room increases
self-awareness? That would imply having an independent way of measuring
self-awareness. Carver and Scheier claimed that the mirrors created a
high-self-awareness condition. They never showed that in fact it did.
They used the assumption as a premise in concluding that there was an
effect of self-awareness on something else. This was an example of
petitio principii.
BP earlier: Simple logical
errors should be edited out, such as petitio principii. The error
of post hoc ergo propter hoc should be caught and eradicated
before publication. I don’t claim immunity: I expect to be called on such
errors when I make them just as anyone else should expect the same if
they want to be taken seriously.
BG: Examples of these mistakes in the literature would be helpful. I
would think they would call at least for a letter to the editor to help
him or her avoid similar mistakes in the future.
See above reference to Carver and Scheier. If you think a letter would do
any good, please send one to their editors.
BP earlier: My insistence on
high correlations has nothing to do with PCT: I never argue that low
correlations are useless simply because PCT correlations are high. My
argument is that low correlations mean poor predictions as well as
inexcusable injustices to people whose lives are seriously affected by
judgements about them made on the basis of false positive or false
negative test results. You know more than most about statistical analyses
– you should be as strongly against them as you are against falling for
the behavioral illusion.
BG: If someone is opposed to a greater awareness of the implications of
statistics, I haven’t heard their arguments. All the emphasis seems to be
on the other side.
You’ve been away from CSGnet when such arguments went on, I guess.
Really, it does take two to have an argument.
I have spoken up on all these
issues for many years, and have experienced opposition because of my
opinions. However, the opposition has often been expressed as if I’m
saying that the problem is lack of belief in PCT, rather than something
that would be a problem if PCT had never been invented. I think we need
to be more clear about why we think something is good or bad
science.
BG: I hope Rick Marken is listening. His view seems to depart radically
from yours. (I like yours better, but who am I to say?)
BP: You’re you. Don’t you have a right to your opinion?
BG: I am beginning to understand
the source of your frustration. Almost nothing in the study of living
systems can live up to your standards.
BP: Yes, that’s right. I have trouble living up to them myself, but at
least I try to.
BG: No wonder you are unhappy.
Here is an informal experiment that seems to most people to have
validity, albeit not to you or Rick I would judge. A flight instructor
said that blame was more effective than praise in improving the ability
of pilots to learn to land airplanes. “Whenever they make a very
good landing, I praised them. The next landing was always worse. Whenever
they botched a landing, I chewed them out. The next landing was always
better.” I suspect that the pilots were trying their best at all
times. (They were controlling a variety of perceptions that they hoped
would bring about a successful landing.) Of course, I have no data to
prove this.
BP: First, I would have to check out the flight instructor’s claims,
especially that “always”. He’s saying that every student pilot
he taught in this way behaved that way. It’s a good yarn, but I don’t
believe it. That claim would arouse considerable skepticism in me (since
I’ve known a few flight instructors), and I’d want to find out if it’s
true before agreeing with it. Perhaps the instructor was wrong every time
he praised a landing, and right every time he criticised it, and the
students taught themselves to land an airplane. That seems to be about
what you’re proposing.
BG: Nevertheless, I suspect that
the flight instructor’s experience could best be explained by reversion
to the mean, and that, in fact, the landings would look pretty much as
they did even if he never praised or blamed. But reversion to the mean is
a a statistical argument that should not be applied to any individual
pilot’s performance if I understand your argument.
BP: I wouldn’t try to predict how any one pilot would fit in with the
statistical population of students. If you forced me to make a
prediction, I would use the population statistics, but I would expect to
predict wrongly almost as often as I predicted right. You mention an
individual pilot’s performance, and that would be very hard to predict
(unless, as the instructor seemed to imply, every single pilot really
responded to the instructor in the same way).
BG: I agree that if the only
behavior worth understanding were tracking experiments, the world would
be a tidier place.Your message seems to be, if you can’t get high
correlations, don’t bother publishing.
BP: Yes, exactly. Why publish garbage just to get tenure? Well, I know
why, of course. As to tracking experiments, there is one demonstration of
a tracking experiment among the 13 demos in my latest book. Clearly, I
don’t think tracking is the only behavior worth studying.
BG: As I said to Rick, that
certainly makes the literature more manageable. However, I can think of
many experiments where low correlations are very informative. I suspect
you could too, if you tried.
BP: I, too, think that low correlations are very informative. They inform
us that we don’t know enough yet to draw a reliable conclusion. They
inform me that if I have to use the hypotheses being tested to predict
individual performance, I can anticipate only slightly more successes
than failures.
By the way, I’ve concluded that you just can’t help the constant sniping,
and I’ve decided that I don’t have to object to it. So feel
free.
Best,
Bill P.