[From Bruce Abbott (970416.1055 EST)]
I'd be very much interested to hear Rick Marken's impression of Richard
Kennaway's paper, "the physical meaning of the correlation coefficient for
bivariate normal distributions." Rick?
Regards,
Bruce
[From Bruce Abbott (970416.1140 EST)]
Richard Kennaway (970415.1650 BST) --
A few years ago, when the subject of correlations came up here, Bill Powers
posted a long message about the magnitude of correlations you need to
actually be useful for prediction. I was sufficiently interested to sit
down and work out the mathematics behind his verbal description. I was
very impressed. If he knows the mathematics, he has a rare gift for
explaining it in English, and if he doesn't, he has an even rarer
mathematical intuition. Anyway, I wrote up the maths in a note available
at ftp://ftp.sys.uea.ac.uk/pub/kennaway/drafts/correlationinfo.\{dvi,tex\}\.
All the maths in it can be found in standard stats textbooks, but I've
never seen it collected together. It's incomplete, in that I repeat
critical remarks I've seen on CSGNET about 20% correlations being published
as meaningful results, but I'm just taking that on trust without any actual
references. Can anyone suggest any? The library here doesn't get JEAB, so
I can't browse through it, and I've no experience in reading such material.
Richard, JEAB is the _last_ place you would want to look for
statistically-based studies, let alone the sort you are looking for that
employ correlational methods and report generally low correlations. JEAB is
the premere journal in psychology for reporting the results of "n-of-1"
(single-subject) experimental studies -- the same focus on the individual
that is emphasized in PCT.
I have read your paper on correlation and find it impressive as it
demonstrates that you have an excellent grasp of the mathematics involved.
A minor quibble: correlations are neither proportions nor percentages
(unlike proportions, they vary between -1.0 and +1.0) and are never reported
as such, although this is perfectly acceptable for the coefficients of
determination (r-squared) and nondetermination (1 - r-squared) to be
reported these ways.
As you note, most of what you present is found in standard texts on
statistics; in fact I found little or nothing new, certainly nothing that
would not be familiar to life-science or social-science researchers
acquainted with correlational methods. That said, you have done a nice job
of laying out the basic logic and interpretation of the Pearson r and its
relationship to the proportion of variance accounted and unaccounted for,
scatter about the "best-fitting" (least squares) straight line, reduction in
the variance of Y when given X, the correct classification of Y given a
knowledge of X, and the reduction in uncertainty X provides about Y. You
have done a fine job on this and I find nothing here to complain about.
One common application of the findings you present is in the evaluation of
the reliability of psychological tests; another is in the assessment of
interobserver agreement. Correlation coefficients below .95 or .90 are
typically judged to indicate poor reliability of the test. A similar
criterion holds for validity, as for example, when using the results of
tests such as the Scholastic Aptitude Test to predict success in college (as
measured by, e.g., grade point average). In the latter case correlations
lower than this yield unacceptable numbers of errors in the decision to
admit students based on the prediction that they will succeed or not succeed.
As you note, Bill P.'s intuition about the ability (or inability) to predict
_individual_ performance based on Pearson r (that is, on linear regression)
is correct: correlations must be very high before reasonably good prediction
of individual cases becomes possible. So far, so good.
But then you go on to interpret your mathematical findings as follows.
Speaking of correlations of, say, 0.50 or under, you state:
Yet because a sufficient quantity of data has been amassed to be statistically
sure that the correlation differs from 0, perhaps even at the 99% confidence
level, such a result can be published even though it is totally meaningless.
The question is, meaningless for what purpose? For predicting a particular
individual's performance (Y) based on that person's value of X, yes,
correlations must be very high indeed to be of any value. But your
conclusion above assumes that those published correlations were published
based on their supposed utility in making such predictions. This is incorrect.
Correlations are most often computed in order to determine whether there is
any linear association between the variables in question. Such associations
may arise through a variety of mechanisms; one rather well-known case in
this forum is through control-system activity when variations in a
disturbance to the CV are negatively correlated with variations in the
action of the system, as in the rubber band demo. Such associations (or the
_lack_ of them where they might be expected) may be important both for
developing and testing specific theories of system organization. Where a
number of variables are involved, all free to vary at least to some extent,
the correlation between any one of them and the response measure may be
poor. Here is an example I worked up involving four predictor variables and
a response variable. Each predictor variable is orthogonal to the others in
the population (though not necessarily in the sample), and each is normally
distributed. The correlations of predictors with the response variable were
as follows:
P1 P2 P3 P4
r 0.519 0.544 0.527 0.468
r-sq 0.269 0.296 0.278 0.219
All of these correlations are within the range of values declared as
"meaningless" in your paper. It may therefore come as something of a
surprise to some readers that the four predictors used together perfectly
predict the value of the response measure. Low correlations are to be
expected when there are numerous uncontrolled variables having an influence
on the response measure and these variables are not highly correlated with
each other. In the life sciences and social sciences, many influential
variables often cannot be experimentally controlled (held constant); in such
cases relatively low correlations will be the rule.
Research aimed at identifying variables having an influence on the response
measure (or, indeed, mutual influences) can be likened to a signal detection
task in which variation in the predictor variable is analogous to the signal
and variation in the response variable is analogous to the signal plus noise
(if the signal has a relationship with the response variable) or noise alone
(if the signal has no such relationship). Viewed in this way, r-sq/(1 -
r-sq) is analogous to a signal-to-noise ratio, in which the "noise" is the
(unmeasured) contribution of other variables to variation in the response
measure. When S/N is low, the signal is difficult to detect and larger
samples are required in order to discriminate the signal from the noise.
To summarize, correlations that individually are "useless" for prediction
may sometimes be combined so as to provide useful predictions (so long as
the correlations are real and not sampling artifacts). In addition such
correlations may prove meaningful within the context of building and testing
a theoretical model of the system under consideration. To conclude that
published results are "meaningless" simply because the reported correlations
are relatively low, one must assume that such correlations can only be used
_individually_ to make point predictions of Y based on a single X, and that
they were gathered and published for this specific purpose. Generally
speaking these assumptions are false.
Regards,
Bruce
[From Bill Powers (970417.0420 MST)]
[From Bruce Abbott (970416.1140 EST)]
Richard Kennaway (970415.1650 BST) --
Richard can speak for himself, but I have a couple of comments:
Correlations are most often computed in order to determine whether there
is any linear association between the variables in question.
I think that adding the modifier "linear" is incorrect unless the
correlation is very high. As Richard shows, a correlation of 0.866 is
required to get one bit of information about Y from knowing X -- meaning
that you can't distinguish between linear and nonlinear relationships if the
correlation is less than that. I suspect that even to distinguish between a
first and a second power relationship would require correlations MUCH higher
than 0.866.
Such associations
may arise through a variety of mechanisms; one rather well-known case in
this forum is through control-system activity when variations in a
disturbance to the CV are negatively correlated with variations in the
action of the system, as in the rubber band demo. Such associations (or
the _lack_ of them where they might be expected) may be important both for
developing and testing specific theories of system organization.
In this case the correlation between disturbance and action has be be very
high, _in addition_ to the correlation with the controlled variable being
very low. The Test does not consist ONLY of looking for a lack of effect.
You must also demonstrate that there is an output from the system that is
opposing the effect of the disturbance. Without that high correlation, there
could be many alternatives to control as an explanation for why the
disturbance fails to affect the proposed controlled variable.
Here is an example I worked up involving four predictor variables and
a response variable. Each predictor variable is orthogonal to the others
in the population (though not necessarily in the sample), and each is
normally distributed. The correlations of predictors with the response
variable were as follows:P1 P2 P3 P4
r 0.519 0.544 0.527 0.468
r-sq 0.269 0.296 0.278 0.219All of these correlations are within the range of values declared as
"meaningless" in your paper. It may therefore come as something of a
surprise to some readers that the four predictors used together perfectly
predict the value of the response measure.
Obviously the response variable was some regular function of the values of
P1 through P4, and there must have been another function constraining the
four values so they were related to each other. Correlations involving
_regular_ functions are meaningless, as I understand it: the regular
function is all you need. If you have Y = sin(X), where X = k*t, and sample
the values at random times (or regular times incommensurate with k/(2*pi)),
you'll end up with a correlation of zero, even though X perfectly predicts Y
once you know the function relating them.
Low correlations are to be
expected when there are numerous uncontrolled variables having an
influence on the response measure and these variables are not highly
correlated with each other.
Low correlations are also to be expected even when there are no uncontrolled
variables, if you are using the wrong model to represent the relationship.
In the case of Y = sin(X), simply to compute a Pearson's r is to assume a
linear relationship between X and Y. In this case, it's using the wrong
model that leads to the low correlation.
In the life sciences and social sciences, many influential
variables often cannot be experimentally controlled (held constant); in
such cases relatively low correlations will be the rule.
That is the usual assumption, I agree. But if the influential variables were
known, they could be measured and accounted for. The problem here is
assuming that the reason for the poor predictions is that there are
_unknown_ variables affecting the result. This is tantamount to claiming
that there is nothing wrong with the theory under which the predictions are
being made, and that if these unknown variables could only be held constant,
the prediction would be perfect. That does not follow at all. This is like
saying that the reason for a failure of reinforcement theory to predict the
application of operant conditioning in the classroom is an unknown history
of reinforcement on the part of each student. That would be the correct
explanation if it were known that the theory is correct. However, it is also
possible that the theory is incorrect, which would lead to the same result.
To summarize, correlations that individually are "useless" for prediction
may sometimes be combined so as to provide useful predictions (so long as
the correlations are real and not sampling artifacts). In addition such
correlations may prove meaningful within the context of building and
testing a theoretical model of the system under consideration.
I thought this would be your point: "A whole lot of bad experiments can be
equal to one good experiment." I don't believe it.
Best,
Bill P.
[From Bruce Abbott (970417.1340 EST)]
Bill Powers (970417.0420 MST) --
Bruce Abbott (970416.1140 EST)
Correlations are most often computed in order to determine whether there
is any linear association between the variables in question.
I think that adding the modifier "linear" is incorrect unless the
correlation is very high. As Richard shows, a correlation of 0.866 is
required to get one bit of information about Y from knowing X -- meaning
that you can't distinguish between linear and nonlinear relationships if the
correlation is less than that. I suspect that even to distinguish between a
first and a second power relationship would require correlations MUCH higher
than 0.866.
Excuse me, but Pearson r is _defined_ as a number indicating the degree and
direction (+ or -) of _linear_ association. It is a number that indexes how
well the points _fit_ a straight-line function (by a least-squares
criterion); this is referred to as the degree of linear association.
Any function can be decomposed into linear, quadradic, cubic, etc.
components, just as any waveform can be decomposed into a set of sine waves
differing in frequency and amplitude. If the "true" function underlying the
data were, for example, a positive exponential viewed over some short range,
it might not be possible to distinguish this upward-curving function from a
linear function with positive slope; in fact the straight line might provide
a good approximation to the exponential over this range. Pearson r would
index how well the linear approximation fit the data. Linear models
generally are so powerful precisely because linear fits often yeild good
approximations over the range of data under consideration.
Such associations
may arise through a variety of mechanisms; one rather well-known case in
this forum is through control-system activity when variations in a
disturbance to the CV are negatively correlated with variations in the
action of the system, as in the rubber band demo. Such associations (or
the _lack_ of them where they might be expected) may be important both for
developing and testing specific theories of system organization.
In this case the correlation between disturbance and action has be be very
high, _in addition_ to the correlation with the controlled variable being
very low. The Test does not consist ONLY of looking for a lack of effect.
You must also demonstrate that there is an output from the system that is
opposing the effect of the disturbance. Without that high correlation, there
could be many alternatives to control as an explanation for why the
disturbance fails to affect the proposed controlled variable.
A somewhat lower correlation may indicate that (a) there are disturbances
acting on the CV which you are not able to measure, (b) the reference level
of your control system is varying during the test (and this information is
not available for inclusion in your simulation), (c) the environmental
feedback function is varying over the course of observation and, again, you
are not able to measure this variation, (d) your model is wrong, (e) other.
Measuring these other sources of variation would of course be the best
solution to the problem, although this is not always possible given
available resources. The failure to observe the expected high correlations
is itself an important fact, though, which would then justify further
investigation. Thus, even low correlations may have scientific utility.
Here is an example I worked up involving four predictor variables and
a response variable. Each predictor variable is orthogonal to the others
in the population (though not necessarily in the sample), and each is
normally distributed. The correlations of predictors with the response
variable were as follows:P1 P2 P3 P4
r 0.519 0.544 0.527 0.468
r-sq 0.269 0.296 0.278 0.219
All of these correlations are within the range of values declared as
"meaningless" in your paper. It may therefore come as something of a
surprise to some readers that the four predictors used together perfectly
predict the value of the response measure.
Obviously the response variable was some regular function of the values of
P1 through P4, and there must have been another function constraining the
four values so they were related to each other. Correlations involving
_regular_ functions are meaningless, as I understand it: the regular
function is all you need.
I am not quite sure what you are talking about here. Yes, the response
variable was some regular function of the values of P1 through P4. No,
there was no other function constraining the four values so that they were
related to each other. As noted in my post, the variables P1 through P4
were unrelated in the population, although spurious near-zero correlations
do appear in the statistical sample. Here are the observed correlations
between the predictors:
P1 P2 P3
P2 0.060
P3 0.009 0.057
P4 -0.031 0.007 0.019
You last sentence has me baffled. The object of observing the correlations
in the data is to account for the variation in the response variable. You
don't know what "regular function," if any, may be at work. The regular
function is all you need? The regular function is what you seek to discover!
Low correlations are to be
expected when there are numerous uncontrolled variables having an
influence on the response measure and these variables are not highly
correlated with each other.
Low correlations are also to be expected even when there are no uncontrolled
variables, if you are using the wrong model to represent the relationship.
In the case of Y = sin(X), simply to compute a Pearson's r is to assume a
linear relationship between X and Y. In this case, it's using the wrong
model that leads to the low correlation.
Yes, of course. That is where modeling comes in, and analytic skill (e.g.,
I can show you a nice correlation between Y and sin(X) over the X range 90
to 270 degrees). But whether correlations are low because there are
influential variables not being measured or because the wrong model (linear)
is being applied to the data, those correlations are still scientifically
informative and useful. Furthermore, correlations should never be
interpreted without a look at the scatterplot. If a strong _non_linear
relationship is present, that will be revealed by the plot.
In the life sciences and social sciences, many influential
variables often cannot be experimentally controlled (held constant); in
such cases relatively low correlations will be the rule.
That is the usual assumption, I agree. But if the influential variables were
known, they could be measured and accounted for.
Yes, and the search for them is what generates those tables of correlation
coefficients, most of which are low because the X-variable is not an
influential variable in the setting being investigated. Having obtained low
correlations, one understands that the current model or variable is not
doing a good job, and launches further investigations to rectify the problem.
The problem here is
assuming that the reason for the poor predictions is that there are
_unknown_ variables affecting the result. This is tantamount to claiming
that there is nothing wrong with the theory under which the predictions are
being made, and that if these unknown variables could only be held constant,
the prediction would be perfect.
The problem you note is true of any scientific investigation: the failure to
predict accurately indicates either that the model is incomplete or that it
is incorrect, or both. My statement is _not_ tantamount to claiming that
"there is nothing wrong with the theory." All I have claimed is that the
failure to identify all the influential variables is _one_ reason why one
might obtain low correlations. Your conclusion would be true only if I had
claimed that this is the _only_ reason why one might obtain low correlations.
To summarize, correlations that individually are "useless" for prediction
may sometimes be combined so as to provide useful predictions (so long as
the correlations are real and not sampling artifacts). In addition such
correlations may prove meaningful within the context of building and
testing a theoretical model of the system under consideration.
I thought this would be your point: "A whole lot of bad experiments can be
equal to one good experiment." I don't believe it.
I don't believe it either.
That is the second time in this post you put words in my mouth that I
neither expressed nor implied. I suppose it's good _polemics_ on your part,
but it certainly does not serve to advance _anyone's_ understanding. In
your "restatement" of my summary, you change "low correlations" into "bad
experiments," and paraphrase me as saying that bad experiments can be
combined to yeild one good experiment. This is nonsense. I didn't say it,
I didn't imply it.
What I _did_ state is that several variables having low correlations with a
response measure sometimes can be combined to yeild a function that predicts
well. As this was _demonstrated empirically_ in the simulation I reported
on in my post, the truth of this statement is beyond question. For this
reason it is simply _wrong_ to assume merely on the basis of low correlation
that a given relationship between two variables is "useless."
I wonder if Richard Kennaway will favor us with a response to my critique of
his paper. Richard, you're not back in "lurk" mode, are you?
Regards,
Bruce
[From Bill Powers (970417.1329 MST)]
From Bruce Abbott (970417.1340 EST)--
I think that adding the modifier "linear" is incorrect unless the
correlation is very high. As Richard shows, a correlation of 0.866 is
required to get one bit of information about Y from knowing X -- meaning
that you can't distinguish between linear and nonlinear relationships if
the correlation is less than that. I suspect that even to distinguish
between a first and a second power relationship would require
correlations MUCH higher than 0.866.Excuse me, but Pearson r is _defined_ as a number indicating the degree
and direction (+ or -) of _linear_ association. It is a number that
indexes how well the points _fit_ a straight-line function (by a
least-squares criterion); this is referred to as the degree of linear
association.
In the section on "Mutual information", Kennaway shows that to predict Y
from X to one part in N (i.e., to distinguish N different values of the
variable), the correlation c must be
c := sqrt(1 - 1/N^2).
To distinguish 2 points (the minimum required to establish a regression
line) the correlation must be 0.866 (for p < 0.05, I presume). But this does
not allow you to assign values on any finer scale. You can say only that if
X is known to be positive, Y will be SOME positive number -- but you can't
say WHAT positive number. That's what "1 bit of Information" means. For 3
points you must have c = 0.94 -- this would enable you to say that if X is
negative, Y is negative, if X is 0, Y is zero, and if X is positive, Y is
positive. To distinguish 5 levels of Y (two negative, two positive, and 0),
you need c = 0.98.
This says to me that drawing a regression line through a scatter plot
representing c = 0.866 is highly misleading; it implies a linear
relationship _between_ the extremes that is unjustified. It takes three
points to establish a quadratic relationship, so you'd need c = 0.94 to get
a _minimal_ indication that Y = X^2 fits the data (you can perform any
transformation you like before applying the "linear" correlation method).
Any function can be decomposed into linear, quadradic, cubic, etc.
components, just as any waveform can be decomposed into a set of sine
waves differing in frequency and amplitude. If the "true" function
underlying the data were, for example, a positive exponential viewed over
some short range, it might not be possible to distinguish this
upward-curving function from a linear function with positive slope; in
fact the straight line might provide a good approximation to the
exponential over this range. Pearson r would index how well the linear
approximation fit the data. Linear models generally are so powerful
precisely because linear fits often yield good approximations over the
range of data under consideration.
Linear models are "powerful" only if you have some independent way of
knowing that the relationship should be linear. When you're presented with a
data set and have no knowledge of what the relationship is, the linear model
(where I come from) is considered the _weakest_ assumption, and it's
generally used only because of a lack of justification for anything more
complicated. And also, no less important, because there aren't very many
nonlinear equations that we can solve!
To distinguish an exponential from a straight line, you need at least two
slopes; i.e., three data points, which implies c = 0.94, at least. And that
would be quite insufficient to distinguish the exponential (A*exp(kx)) from
a quadratic (Ax^2 + Bx).
You can't get away from the fact that with a correlation of only 0.866, you
could not distinguish a straight line from an exponential relationship by
ANY means. But I should let Richard have his say on that.
......................
A somewhat lower correlation may indicate that (a) there are disturbances
acting on the CV which you are not able to measure, (b) the reference
level of your control system is varying during the test (and this
information is not available for inclusion in your simulation), (c) the
environmental feedback function is varying over the course of observation
and, again, you are not able to measure this variation, (d) your model is
wrong, (e) other.
Measuring these other sources of variation would of course be the best
solution to the problem, although this is not always possible given
available resources. The failure to observe the expected high
correlations is itself an important fact, though, which would then justify
further investigation. Thus, even low correlations may have scientific
utility.
You miss the point. Even to see that there are significant deviations from
theory, you must have very good data; the basic correlations must be in the
high 0.9s. To distinguish among different explanations for the deviations,
you must have better data still, or a better model. This means that the
major part of your labor has to go into getting very good data and improving
the model, not into devising ever more "powerful" ways of extracting
information from noisy data.
............................
Here is an example I worked up involving four predictor variables and
a response variable.
...
I am not quite sure what you are talking about here. Yes, the response
variable was some regular function of the values of P1 through P4. No,
there was no other function constraining the four values so that they were
related to each other. As noted in my post, the variables P1 through P4
were unrelated in the population, although spurious near-zero correlations
do appear in the statistical sample. Here are the observed correlations
between the predictors:P1 P2 P3
P2 0.060
P3 0.009 0.057
P4 -0.031 0.007 0.019You last sentence has me baffled. The object of observing the >correlations
in the data is to account for the variation in the response variable. You
don't know what "regular function," if any, may be at work. The regular
function is all you need? The regular function is what you seek to >discover!
Bafflement all around, here. It might help if you told us what the trick is!
...........................
Low correlations are to be
expected when there are numerous uncontrolled variables having an
influence on the response measure and these variables are not highly
correlated with each other.Low correlations are also to be expected even when there are no
uncontrolled variables, if you are using the wrong model to represent the
relationship. In the case of Y = sin(X), simply to compute a Pearson's r
is to assume a linear relationship between X and Y. In this case, it's
using the wrong model that leads to the low correlation.Yes, of course. That is where modeling comes in, and analytic skill
(e.g., I can show you a nice correlation between Y and sin(X) over the X
range 90 to 270 degrees).
Well, not THAT nice! What is it, by the way? Not that it would mean anything
-- the distribution is not normal.
But whether correlations are low because there are
influential variables not being measured or because the wrong model
(linear) is being applied to the data, those correlations are still
scientifically informative and useful. Furthermore, correlations should
never be interpreted without a look at the scatterplot. If a strong
_non_linear relationship is present, that will be revealed by the plot.
Not with a correlation of only 0.866. Unless you mean a VERY strong
nonlinearity or an obviously non-normal distribution.
In the life sciences and social sciences, many influential
variables often cannot be experimentally controlled (held constant); in
such cases relatively low correlations will be the rule.That is the usual assumption, I agree. But if the influential variables
were known, they could be measured and accounted for.Yes, and the search for them is what generates those tables of correlation
coefficients, most of which are low because the X-variable is not an
influential variable in the setting being investigated. Having obtained
low correlations, one understands that the current model or variable is
not doing a good job, and launches further investigations to rectify the
problem.
I think we agree on that. The point of disagreement between us is what
constitutes "rectifying" the problem.
....
The problem you note is true of any scientific investigation: the failure
to predict accurately indicates either that the model is incomplete or
that it is incorrect, or both. My statement is _not_ tantamount to
claiming that "there is nothing wrong with the theory."
I'm not saying that YOU would do this. But it is commonly done, simply by
failing to mention the alternative to "uncontrolled variables" as an
explanation of the scatter. How many papers have you seen in which the
author says that the low correlation can be explained either by uncontrolled
variables, or by the possibility that stimuli do not cause responses?
I thought this would be your point: "A whole lot of bad experiments can
be equal to one good experiment." I don't believe it.I don't believe it either.
That is the second time in this post you put words in my mouth that I
neither expressed nor implied. I suppose it's good _polemics_ on your
part, but it certainly does not serve to advance _anyone's_ understanding.
I was thinking back to your remarks about bad experiments in EAB, in which
you said quite clearly that even though each experiment might have left much
to be desired, when many experiments are done, all indicating about the same
results, the net effect can be better than the results of any one
experiment. The problem is in assuming that all these experiments yield
about the same results, when in fact each one could be subject to many
different interpretations, because of the incomplete or wrong observations.
If everyone doing these experiments agrees on an interpretation within the
range covered by the experiments, this merely shows a desire to reach
agreement; it by no means shows that the data especially support that
particular interpretation.
In your "restatement" of my summary, you change "low correlations" into
"bad experiments,"
I think that low correlations are generated by bad experiments, so in my
view I didn't change anything. I don't expect people to suddenly start
getting high correlations by improving their bad experiments; I just wish
they would stop PUBLISHING before they have refined their experiments.
What I _did_ state is that several variables having low correlations with
a response measure sometimes can be combined to yeild a function that
predicts well. As this was _demonstrated empirically_ in the simulation I
reported on in my post, the truth of this statement is beyond question.
I'd still like to see what's behind those numbers. Is this a common
situation? If not, how do you tell when you're dealing with an example of it?
For this
reason it is simply _wrong_ to assume merely on the basis of low
correlation that a given relationship between two variables is "useless."
I agree that it's wrong to assume that the correlation couldn't be improved
by a better understanding of the underlying processes. But the fact is, a
correlation of 0.866 permits you only to predict the _sign_ of an effect,
which is useless if you need to predict more than that.
Best,
Bill P.
[From Rick Marken (970417.2200 PDT)]
I have not yet read Richard Kennaway's paper but from what I can
glean about it from the exchange between Abbott and Powers,
it seems to be a _masterpiece_.
I found this exchange to be particularly illuminating.
Bruce Abbott (970417.1340 EST) --
Measuring these other sources of variation would of course be the
best solution to the problem, although this is not always possible
given available resources. The failure to observe the expected
high correlations is itself an important fact, though, which
would then justify further investigation. Thus, even low
correlations may have scientific utility.
Bill Powers (970417.1329 MST) --
Even to see that there are significant deviations from
theory, you must have very good data; the basic correlations
must be in the high 0.9s. To distinguish among different
explanations for the deviations, you must have better data
still, or a better model. This means that the major part of
your labor has to go into getting very good data and improving
the model, not into devising ever more "powerful" ways of
extracting information from noisy data.
I think Bill makes an _extremely_ important point here. When I was
a conventional psychologist I delt with noisy data by "devising ever
more "powerful" ways of extracting information from" it. I can
understand the attraction; you get to use all kinds of cool statistical
techniques, you feel like you are doing some very brilliant analyses
and, most important, you don't have to go
through the labor of collecting more data using different
procedures and techniques, with no guarantee that you will ever
get high quality results.
When I became a PCT psychologist I started doing research in a new way.
If the data from a PCT experiment were noisy, I didn't try to make sense
of them with powerful statistical techniques. Instead, I would try to
change the experiment so that I would get good data
or I would try to change the model to get a better fit. It turns out
that this is the way "real" scientists go about doing their research.
But I didn't start doing research this way because I wanted to
imitate real scientists. I did it because the PCT model works so well.
The results of a well conceived PCT experiment are clear and precise and
perfectly consistent with the PCT model. So when the results of
a PCT experiment are noisy, one's inclination is not to blame extraneous
variables or "the inherent variability of behavior" or whatever (as in
conventional psychology); that is, one's
inclination is not to blame the phenomenon. Rather, one's
inclination is to blame oneself.
If I am not able to design an experiment so that it produces clear
and consistent results, I give up on the experiment until a cleverer
person comes along and does it right. If I were still a conventional
psychologist, instead of giving up on an experiment that produced noisy
data, I would do some fancy statistics on the data and publish the
results. Of course, if I were a conventional psychologist I probably
wouldn't find much of value in Richard Kennaway's paper either.
By the way, a good example of what I think is the right way to go about
doing behavioral research is described in Dick Robertson's recently
posted "Testing the self as a control system". Dick and
his colleagues kept changing their test procedure until they got fairly
clear, noise free evidence of control of a rather complex
perceptual variable (which can be called "self image"). Nice work, Dick.
Best
Rick
[From Bruce Abbott (970418.1100 EST)]
Rick Marken (970417.2200 PDT) --
I found this exchange to be particularly illuminating.
Bruce Abbott (970417.1340 EST) --
Measuring these other sources of variation would of course be the
best solution to the problem, although this is not always possible
given available resources. The failure to observe the expected
high correlations is itself an important fact, though, which
would then justify further investigation. Thus, even low
correlations may have scientific utility.Bill Powers (970417.1329 MST) --
Even to see that there are significant deviations from
theory, you must have very good data; the basic correlations
must be in the high 0.9s. To distinguish among different
explanations for the deviations, you must have better data
still, or a better model. This means that the major part of
your labor has to go into getting very good data and improving
the model, not into devising ever more "powerful" ways of
extracting information from noisy data.
I think Bill makes an _extremely_ important point here.
I agree with Bill's point, as I stated before. And I make an extremely
important point here as well, although you have chosen to ignore it. Both
Bill's position and mine are correct, but refer to different points in an
investigation. The correlational approach is useful during the early stages
of an investigation, when one is attempting to identify what variables are
involved and how they interact. At this stage, even relatively low
correlations can be informative. Later, when you have created a model or
models suitable for testing, very high correlations are desirable as
indicating a good "fit" of model to data, and to discriminate alternative
models.
When I was
a conventional psychologist I delt with noisy data by "devising ever
more "powerful" ways of extracting information from" it. I can
understand the attraction; you get to use all kinds of cool statistical
techniques, you feel like you are doing some very brilliant analyses
and, most important, you don't have to go
through the labor of collecting more data using different
procedures and techniques, with no guarantee that you will ever
get high quality results.
When you were a conventional psychologist you must have been enthralled with
statistics. I don't view this sort of activity as productive: if your data
are messy, you can't save the study with "ever more powerful ways of
extracting information from it." This is _not at all_ the approach I am
talking about. What I am talking about is that a set of relatively low
correlations among a number of measured variables can turn out to be useful
for a number of scientific purposes. As I've already gone over this in
another post, I won't repeate it here. But please, if you are going to
offer "replies" to my posts, let's get on the same frequency. To pretend
that I am talking about one thing when I am discussing something else
entirely, and to offer criticism on that basis, contributes nothing to
understanding.
When I became a PCT psychologist I started doing research in a new way.
If the data from a PCT experiment were noisy, I didn't try to make sense
of them with powerful statistical techniques. Instead, I would try to
change the experiment so that I would get good data
or I would try to change the model to get a better fit. It turns out
that this is the way "real" scientists go about doing their research.
What a relief: this is the way _I_ was taught to do scientific research.
Try reading Murray Sidman's (1960) _Tactics of Scientific Research_ for an
excellent presentation of this method.
The results of a well conceived PCT experiment are clear and precise and
perfectly consistent with the PCT model. So when the results of
a PCT experiment are noisy, one's inclination is not to blame extraneous
variables or "the inherent variability of behavior" or whatever (as in
conventional psychology); that is, one's
inclination is not to blame the phenomenon. Rather, one's
inclination is to blame oneself.
When the results of a PCT experiment are noisy, there is no use "blaming"
anyone. One looks for an explanation for the noisy data. This explanation
must _necessarily_ involve either extraneous variables or problems with the
structure of the model being fit to the data. It is not to be found in the
"self."
If I am not able to design an experiment so that it produces clear
and consistent results, I give up on the experiment until a cleverer
person comes along and does it right.
Well, that explains why PCT has not advanced beyond simple tracking
experiments in over 25 years. You leave all the really hard work to others
-- and there ain't no others.
If I were still a conventional
psychologist, instead of giving up on an experiment that produced noisy
data, I would do some fancy statistics on the data and publish the
results.
Of course, if I were a conventional psychologist I probably
wouldn't find much of value in Richard Kennaway's paper either.
My, how you do go on about a paper you haven't even read. (I don't suppose
we can rely on you for an _objective_ evaluation, can we?) But more to the
point: I have already stated (twice now, I believe) that Kennaway's paper is
right on the mark as far as it goes. No statistically competent
"conventional psychologist" would find anything in it objectionable, insofar
as its conclusions about the ability to predict the Y-value of a specific
point from its X-value are concerned. You imply (again, without having even
seen the paper!) that the paper has something important -- and disturbing --
to say about the _general_ usefulness of moderate correlations (disturbing,
that is, to conventional psychologists). I don't know how you can reach
that conclusion without having read the paper. If you still hold that view
_after_ reading Kennaway's paper, well, we'll cross that bridge when we come
to it.
Regards,
Bruce
[Martin Taylor 970419 13:50]
Bill Powers (970417.1329 MST)
To distinguish 2 points (the minimum required to establish a regression
line) the correlation must be 0.866 (for p < 0.05, I presume). But this does
not allow you to assign values on any finer scale. You can say only that if
X is known to be positive, Y will be SOME positive number -- but you can't
say WHAT positive number. That's what "1 bit of Information" means.
No it doesn't. It means that the variance of estimate has been cut to
1/4 of its original value (Standard deviation halved--uncertainty reduced
by -log2(0.5)). Only if you have values limited to belonging to a discrete
set of distinct values (e.g. the integers) can you say what you said.
For the rest, I think Bruce Abbott gave a sufficient response.
But I think a little expansion on his four-variable demonstration might be
in order. In a deterministic universe, if we knew all the influences that
affect a particular variable, and knew the values of all of them, we would
be able to predict the value of the variable exactly. Omit any of the
influences, and the prediction becomes uncertain--for replicated values
of the rest of the influences, the value of the variable changes because
of differences in the value of the unmeasured influence. In Bruce's micro-
universe, the variable was equally influenced by four independent variables.
If one knew all the values of those variables and the nature of the
influence, one would be able to predict the value of the variable with
no error.
Now suppose you were an experimenter, looking to see what was affecting this
variable that interested you, and you guessed that Bruce's influence P1
might be doing so. You might vary P1 and see what effect it had on the
variable V. You get a correlation with r^2 about 1/4. If you say "That's
useless. I'll drop consideration of P1 because the effect is too uncertain
to mean anything" you lose your chance of ever finding out how V is caused.
But if you keep V and P1, and now look at what you can do about V having
now guess that P2 also has an influence, you find another lowish correlation.
Now you find that P2 has a r^2 of about 1/4 with V, but of about 1/3 with
the variation of V after the effect of P1 _on each individual trial_ has
been partialled out. And so on with P3, which has an r^2 of about 0.5
after the "useless" effects of P1 and P2 have been applied to the
individual trials, and finally P4 has an r^2 of 1.0--but _only_ if you
take account of the effects that you had deemed to have uselessly low
correlations. If you don't take account of those "trivial" influences,
P4 also seems to have a trivial effect, with r^2 = 1/4.
What's useless depends on what you want to use it for.
Martin
For this
reason it is simply _wrong_ to assume merely on the basis of low
correlation that a given relationship between two variables is "useless."I agree that it's wrong to assume that the correlation couldn't be improved
by a better understanding of the underlying processes. But the fact is, a
correlation of 0.866 permits you only to predict the _sign_ of an effect,
which is useless if you need to predict more than that.
No. It enables you to predict the _value_ of the correlated variable twice
as accurately as you could otherwise have done.
That's what's meant by the difference between _some_ information and none.
And between some information and complete information, which would allow
you to predict the value precisely.
Martin
[From Bill Powers (2007.07.22.1235 MDT)]
Here is the Kennaway paper on predicting individual data from group data.
Bill P.
No virus found in this outgoing message.
Checked by AVG Free Edition.
Version: 7.5.476 / Virus Database: 269.10.10/908 - Release Date: 7/19/2007 6:10 PM
StatPredictJRK.pdf (250 KB)
[David Goldstein}
Bill,
Do you know if this paper by Kennaway was actually published?
I imagine that it would have made a big splash.
The Psychological test with the highest test-rest reliability are the IQ
tests by David Wechsler.
I recall that they are about .95.
from Kenneways paper, using this test to predict what an individal who
takes it at time 1 will do
at time 2 will not be very accurate. Using Table 3, the probability that
it will fall within plus or minus
a half a decile will be .47. In the IQ test, a standard deviation is 15
IQ points. If plus or minus
2 standard deviations on each side of the mean of 100 covers most of the
range of 60 IQ points, a decile would be
6 IQ points; a half of a decile would be 3 IQ points. So the probability
that the person IQ would fall
within plus or minus 3 IQ points would be .47. This is about what the
manual says.
The maximum possible correlation between two tests is equal to the
square root of their respective test
re-test reliabilities. So we would need a test of which is at least as
reliable as the IQ test to get any meaningful
results for an individaul.
David
ยทยทยท
-----Original Message-----
From: Control Systems Group Network (CSGnet)
[mailto:CSGNET@LISTSERV.UIUC.EDU] On Behalf Of Bill Powers
Sent: Sunday, July 22, 2007 2:36 PM
To: CSGNET@LISTSERV.UIUC.EDU
Subject: Kennaway paper
[From Bill Powers (2007.07.22.1235 MDT)]
Here is the Kennaway paper on predicting individual data from group
data.
Bill P.
No virus found in this incoming message.
Checked by AVG Free Edition.
Version: 7.5.476 / Virus Database: 269.10.12/910 - Release Date:
7/21/2007 3:52 PM
No virus found in this outgoing message.
Checked by AVG Free Edition.
Version: 7.5.476 / Virus Database: 269.10.12/910 - Release Date:
7/21/2007 3:52 PM