[Martin Taylor 970304 11:00]
Phil Runkel on 3 March 97.
Some time ago some people on the CSGnet were proposing various
numerical indicators of the closeness of a trace of human
behavior to a trace of model behavior. The motivation was that a
correlation tells the extent to which one trace parallels or
makes the same bends as another, but it does not tell the
distance between the traces.
As a main contributor to that discussion, my motivation was a bit
different. If there is control of a certain precision, as assessed
by the stability factor, the correlation, or whatever parameter one
uses, there is a multitude of possible control system structures that
could achieve this precision. The stability factor (etc., etc.) shows
_that_ control is happening, but not _which control structure_ is a
better model than which other that has the same degree of fit.
You can think of the trace or profile
produced by the human as a single point in a multidimensional
space, and of the profile produced by the model as another point
in that same space. The number of dimensions is the number of
pairs of points that are compared in the run of the experiment.
Yes, that's the approach I used. Then my idea was to compare three
points in this space, not two. One is the "ideal" of perfect control,
one the actual human trace, and one the simulation trace. We have
chosen the simulation model as the best fit to the human data that
we can get by varying the parameters for control systems of that
structural type. This means that we are concerned with possible
control systems of other structural types that would fit equally well.
But the simulated control system is just that, a _control_ system. So
some part of the fit will be due to the fact that the human controls,
and the simulation controls. The question is how much of the fit is
due to that fact, and how much is due to the simulation having a
structure that matches the human's control structure reasonably
well.
Now we think about the layout of points in this high-dimensional space.
Two observations apply: (1) the higher the dimensionality, the more
likely it is that two vectors chosen at random will be very close to
orthogonal, and (2) we are dealing with only three points, so we can
always fit a plane--a two-dimensional flat surface--through them. We
are therefore dealing with a triangle and the angles of the triangle.
Call the corners of the triangle C (centre--the ideal), H (human) and
S (simulation). High dimensionality means having many _independent_
sample points. If the control bandwidth is, say, 5 Hz, there are 10
independent sample points per second, no matter how many measurements
are made.
Let's consider extreme cases:
(1) The structure of the simulation is unrelated to that of the human,
and the fit is entirely due to the fact that both the human and the
simulation control equally well. The possible structures for which the
best fit is exactly this good provide data that fall on a circle around
point H. If the fit is largely due to the fact that both human and
simulation control, then the point S will be closer to C than to H, and
in the extreme will be directly between H and C. The simulation will
control too well. The angle CHS approaches zero.
(2) The structure of the simulation is perfect so far as it goes (by which
I mean stuff like being a one-level model, or not having adaptive gain
when the human does--if the human does), and mismatch is due to
things like mouse-slips, momentary failures of attention, and other
things not modelled. Now we can say that the simulation controls because
it has the structure of the human being modelled (so far as it goes)
and the human controls. If the human didn't control, neither would the
simulation, in this case. Of all the possible structures we might have
chosen equally like the human structure, some will control better than
the human, some worse, but we can't say anything about those, since we
don't know what they are. In this case, the angle CHS approaches 90 degrees.
(3) There's a rather absurd possibility, but it has to be included, and that
is that the simulation fits the human data _in spite of_ the fact that the
human controls. This case corresponds to values of the angle CHS appreciably
greater than 90 degrees. If this case ever showed up, I'd worry about the
programming, the modelling, or something. Occasional high values of CHS
are to be expected simply from statistical considerations, but for long
tracks they should be _very_ rare.
All of these cases depend on there being a distance measure of some kind,
whether it be the RMS measure you propose, a correlation measure, or some
other measure of fit. The key is to measure the angle CHS. The noisier
the data, the nearer that angle will be to 45 degrees, where the data are
equally due to the fact that the simulation control and to the fact that
it controls the same way that the human controls (that's an intuitive
statement, not an analytic one, and it could well be wrong).
To recap: values of angle CHS near zero imply inappropriate structures
for the simulation, values near 90 degrees imply good structures, values
much greater than 90 degrees should lead to soul-searching on the part of
the programmer or the analyst, and values near 45 degrees should suggest
that the data may be poor (or that the structure lacks something but also
has something right about it).
Martin