[Martin Taylor 960913 10:15]
Rick Marken (960912.2200)
The idea that "ANY control system that controls accurately will provide a
good fit to the human data" seems to be based on the notion that good
control is simple and that poor control is complex;
No. It's based on the idea that when H(human) and T(target) are very
close and Y(Model) and T are very close, then Y is close to H. That has
nothing to do with simplicity. If P, Q, R, S, are also close to T, then
they are also close to H and it is very hard to determinine whether Y
is the better model of H or whether P, Q, R, or S might be better.
The output of a poor controller is all over the
place; it's a good model, indeed, that can explain all the human
controller's apparently random output variations.One problem with this idea is this: suppose you see a person who is
apprently controlling poorly; the controlled variable is moving all over
the place and outputs are only roughly (negatively) correlated with the
disturbance. The problem is that you have no idea whether the variance in
output (and input) is a result of poor control or good control relative to
a changing reference for the controlled variable.
If H is not close to T, _of course_ it might be because of changing
reference values. Do you think this is the normal case when people "appear"
to have trouble tracking accurately. It isn't because fast-changing
disturbances are harder to track, but because when the disturbance changes
fast, people change their reference values fast in order not to be seen to
be tracking well?
That's a possibility, I suppose, and I would judge it to be a quite likely
possibility with a subject named RM 
But not with a normal person as a
subject? One usually expects one's subjects to be trying to cooperate
(especially when one is one's own subject).
Regardless of whether tracking errors are due to changing references or
to difficulty of control, nevertheless if H is far from T and Y is close
to H, then P, Q, R, S can be far from T _and_ far from H, and you can tell
easily which model is better.
This is true even if the model Y includes the notion that it is changes in
the reference value that cause the tracking error. Nothing in what I have
said relates to the content of the model.
I suggest that the way to compare control models is not to make behavior
noisy but to create situations where the models make different predictions.
That is what I was trying to point out.
---------------more on sleep data analyses---------------
Incidentally, I mentioned yesterday that I had toyed with but not followed
up on a criterion of the kind
D = sqrt((sum(model-actual)^2)/(sum(model error)^2 + sum(actual error)^2))
In going back over what remains of the sleep study analyses after the disk
crash, I found that I had actually computed F = 1-D in at least one of the
analyses (actually, I didn't take the square root of the ratio, so it's F =
1-D^2). F has a maximum value of 1.0, and if the model is a random model
that tracks as well as the human, but using a different structure, F should
take on the value 0.0.
When the F criterion is compared with the correlation criterion, the
two criteria give somewhat different answers. I have the F criterion for
the different tasks over time for one subject for whom I also have the
correlation data, and I think I could recover both criteria for all the
subjects. This is subject 111, for those of you who have access to the raw
data.
For this one subject, there seems to be no correlation across time-blocks
between the F and correlation criteria within each task, but the tasks
give substantially different values for both criteria. The five tasks
were (A) pursuit tracking of a vertical bar target using a vertical bar cursor
below it on the screen , (B) compensatory tracking with the same visual
configuration (i.e. the target stayed fixed and the disturbance was applied
to the cursor location), (C) a small disk travelled around the circumference
of a large circle, with the disturbance pushing the small disk inward and
outward, (D) a pendulum swung, and below it a disk tracked along a circular
path just beyond the bob, the mouse controlling the velocity of the disk
while the disturbance affected the pendulum velocity, (E) A number was
displayed on the screen in large numerals, and the disturbance added to or
subtracted from the number while the subject tried to keep it at "50".
The approximate average values for the two criteria for this subject over
the pre-drug (no sleep loss) period were
task corr F
A 0.965 0.40
B 0.87 0.83
C 0.965 0.65
D 0.945 0.64
E 0.96 0.65
Notice that any F value above zero means that the model fits better than
a random equally good controller. It tells you something different from
what the correlation value tells you. Look particularly at the results for
the B (compensatory tracking) task, which shows the lowest correlation between
human and model, but the best fit by the RMS deviation criterion.
I don't know what all this means, if anything. But I think it might be worth
looking at in the data for the other subjects, and perhaps it might be worth
trying this criterion when looking for optimally fitting parameter values,
too. But we won't know until it is tried, which I may or may not do.
Martin