[From Bill Powers (2001.10.17.0704 MDT)]
Martin Taylor 2001.10.16.15:09 --
Welcome back, Martin.
I raised this question a few years ago, but it was never resolved
effectively.
I am trying to fit a simple control model to real data--it doesn't
matter what; this question is better treated in the abstract. The
"classical" approach is to adjust the model parameters until the
modelled data correlate maximally with the experimental data.
Ideally, if the human-to-target correlation is, say, 0.93, we would
want the model-to-human correlation to be greater than 0.93 and also
greater than the model-to-target correlation. We also want similar
relationships among the RMS difference values. But I find in my
modelling attempts that these do not go together very well. By
varying the model parameters, I can get the usual high correlations
between model and human, but very often for those parameter values
the RMS difference between human and model is large. Parameter values
that improve the RMS ratio (human-model)/(target-model) often give
poor values of the correlation comparison (human-model
correlation)/(target-model correlation).
I noted the same problem some years ago. As you say, we want the model to
behave like a human being, but it's difficult to distinguish such a model
from a perfect controller, because human beings come so close to being
perfect controllers (within a frequency band from zero to one hertz).
I originally used the correlation measure because I was presenting data to
psychologists, and they were used to seeing that sort of measure. For my
own purposes, I have always thought of RMS error as more appropriate, that
being a direct measure of how well the model predicts the actual behavior.
The best measure is a moment-by-moment comparison because it shows in
detail how the model's behavior differs from that of the human being. My
models, for example, typically show less high-frequency tracking error than
a human being does. Adding a perceptual lag to the model partly remedies
that shortcoming. I suspect that adding nonlinearities will also help, but
have never experimented with that. In some cases, differences between model
and person were due to the person's still learning to control better -- the
model assumes constant parameters during a run, but while learning is still
incomplete there can be significant changes even during a one-minute run.
When runs are too long, fatigue and inattention can also cause performance
changes during the run; the model, of course, does not fatigue nor can its
attention wander. (Attention is another question for research, of course).
Of course some differences are just due to noise in the person that is not
in the model (see below).
A sensitivity analysis would reveal the true nature of the problem. The
fact is that the performance of a control system with reasonably high gain
is simply insensitive to variations in many system parameters, particular
the parameters of the output function. This is one reason why the control
strategy has a large evolutionary advantage over any plan-and-execute
strategy. But it makes determination of the real parameters difficult when
accuracy of control is used as the measure of performance. For a control
system with a loop gain of 1000, a 50% drop in output gain would make the
error double -- but the output would change from 0.999 of the disturbance
to 0.998 of the disturbance, which would be experimentally undetectable.
Visual-motor tracking is actually at least that accurate at low
frequencies, being limited mainly by visual acuity when disturbances change
only slowly. This says, by the way, that matching the _errors_ would be
much more sensitive to parameter changes than matching the _outputs_.
On the other hand, small changes in performance can reflect quite large
changes in system parameters, so if you're investigating, say, the effect
of drugs or fatigue on performance, any small performance change you can
reliably see indicates a large change in the behaving system. Just ask how
much a model parameter would have to change to have the same magnitude of
effect.
It's easy enough to see geometrically what is happening here. But it
does not inspire confidence in the use of correlation alone as a
figure of merit for fitting control models to real data. My question
really points in two directions: (1) What is a better figure of merit
than simple correlation to optimize when adjusting parameters for a
model of fixed structure, and (2) is there any background literature
or understanding that would suggest structural modification based on
the way the fitting surfaces in RMS and correlation relate to one
another.
I strongly recomment against using correlation as a figure of merit. It's
mainly useful as a way of contrasting the results of control experiments
with the kinds of experimental results usually obtained in the behavioral
sciences (at the St. Louis meeting, we were regaled with a report of an
explanation that accounted for 7% (and I typed that very carefully -- seven
per cent) of the variance).
You might find it useful to measure RMS errors as a function of frequency
(in the analysis, not during the experiment). Just compute the prediction
error as a function of time and then do a Fourier analysis of it. I would
predict a curve that starts close to zero and rises with frequency.
Another profitable analysis would be to test the RMS error for randomness.
If it's completely random, this would say that the person simply contains a
noise source that is absent from the model. I know that my own error traces
contain a periodic component (see the paper in Hershberger's book), but
that probably reflects a small genetic tremor I have always had. If the
errors are random, there wouldn't be much hope of making a model generate
the _same_ random variations. You could always add a random noise source,
of course, but not the same one.
In other
words, a correlation of 0.95 between model and human tells very
little if the correlation between human and target tracks is 0.97,
unless the RMS distance between human and model is substantially less
than between target and model (which would indicate that human and
model make the same kinds of error).
This was one point of the original "Models and their worlds" paper, which
is going to be published as part of Phil Runkel's forthcoming (some day)
book. If you're testing whether you have the right _kind_ of model, this
sort of match is highly significant -- when alternative models show
correlations of 0.2 or 0.5 with the measured output. When fits are very
bad, RMS error tends toward a limit of 100%, and the correlation measure is
probably more sensitive!
I'm a little dubious about trying to devise a single "figure of merit."
Control is a multidimensional phenomenon, and while a model may predict
very well in one dimension it may not do so well in another. It's sort like
trying to find a single figure of merit with which to compare Einstein and
Horowitz.
Best,
Bill P.