[From Bill Powers (2001.01.11.0828 MST)]
Point 1: Something occurred to me that ought to have been
self-evident years ago: if the control system is built of linear
components (which includes integrators and differentiators), its
output can contain only frequencies that exist in the reference
signal or in the disturbance (apart from any natural oscillatory
frequencies inherent in its own structure).
True!
It follows, then, that it is pointless to try to fit a linear model's
behaviour to that of a human outside the frequency band occupied by
the reference and/or disturbance (unless you are trying to fit a
natural oscillation such as is evident in Parkinson's disease). The
best-fit linear model must be the model that best fits the human data
_only_ within the bandwidth of the externally applied signals. This
is true no matter how complex and how many control levels are
involved.
This is true also, but "bandwidth" is an elastic concept. In most real
systems, there is no sudden upper frequency cutoff; instead, at some
frequency the response begins to fall off at some rate -- a factor of two
per octave, for a system containing one integrator. This means that the
loop gain drops gradually as frequency rises, as well as entailing a phase
lag that increases with frequency. The "corner frequency" traditionally
used to measure bandwidth is the frequency where the amplitude response has
fallen to 0.707 of its low-frequency value. This means that the loop gain
is still half as great as its low-frequency value at twice the corner
frequency, and a quarter as great at four times the corner frequency.
In a simple integrating control system, the corner frequency of the output
function might be, say, 2.5 Hz, but with enough loop gain the closed-loop
response could be flat to 10 Hz or more. If there are significant lags, of
course, the loop gain is limited to smaller values and the closed-loop
response will be correspondingly limited (this is closely related to one of
your favorite subjects, the Nyquist sampling criterion -- I don't mean to
lecture you about something you know quite well).
If the human data contain frequencies outside the band of the
externally applied signals, it must be because of some nonlinearity
in the human. This includes such elements as data-dependent gain
changes, attention shifts (however they might be modelled, they must
involve changes in the gain of a variably-attended control unit), and
so forth. One should not attempt to fit those aspects of human
behaviour with a classical multi-level control system composed of
linear elements.
Agreed, although the amount of error remaining to be accounted for after
fitting to a linear model could be quite small. I'd say the importance of
introducing nonlinearities into the model would increase if the real
behavior showed striking departures from the model behavior.
Point 2, which I have mentioned several times over the years: If the
human controls well, any control system that also controls well will
behave very like the human. The interesting aspects of the model fit
are the ways in which the human deviates from perfect control. One
possible "figure of merit" then is the accuracy with which the
model's deviation from perfect control matches the human's deviation
from perfect control.
Yes, I've been saying that, too, for several decades. This is why I have
recommended using disturbances that are sufficiently difficult to produce
about a 10% RMS error in tracking studies. This makes control by the real
person deteriorate enough that a perfect control model with very high
integral gain and zero lag will not fit the behavior as well as a model
with finite gain and lag will fit it.
The way this shows up in fitting models to behavior is that for each
parameter, there is a best-fit value, with the fit becoming clearly worse
for values either higher or lower than the optimum. Perfect control
results in general from high gain and zero lag, so the model's fit becomes
worse as the parameters bring it closer to perfect control -- when the gain
is too high and the lag is too low.
If the difficulty of the control task is too low, the person's control will
come closer to perfect control, and the sensitivity of the fit to parameter
changes will become lower. With an easy enough task (small and slow
disturbance), the system noise swamps the changes of fit due to changing
parameters, and the best-fit model can't be distinguished from the perfect
control model. This may suggest the kind of "figure of merit" you're
looking for. The best-fit control parameters can be given error bars (say,
2-sigma), and the narrower the bars, the better the model.
In doing this modelling, I have used not only the gain, lag, and
relative prediction gain parameters, but also a power-law
nonlinearity applied to the error signal before it is applied to the
output integrator. In the few examples I have so far tested, the
optimum power is usually between about 1.3 and 1.8, which suggests
that an inherent nonlinearity might exist. At least for the graphic
data, I probably should also do a logarithmic transform of the
perceptual input, but I haven't done that yet, and this might alter
the fit of the power law component.
By "prediction gain" are you referring to first-derivative gain?
I've always thought that introducing nonlinearities would be a good idea,
and I still do -- congratulations on being the first (as far as I know) to
get out of his armchair and actually do it. The power-law approach is a
good one, giving you in effect a one-parameter nonlinearity to adjust. If
you still have residual non-random errors of fit, you might consider a
polynomial, in which you can adjust the contribution of different powers
individually. Second-power and third-power nonlinearities are especially
important -- symmetrical around zero, and non-symmetrical. Muscle
preparations, apparently, have an exponential force output response to
motor signal frequency, although a square law fits about as well.
I haven't heard of other model fits that introduced this kind of
nonlinearity into the connection between the comparator and the
output integrator, and I was wondering if anyone had tried it--and if
so, with what result?
As far as I know, you're the first. My Little Man model is very nonlinear,
but I haven't fit it to real behavior yet, and may never do so (a certain
amount of expensive instrumentation is required, not to mention an interest
in using it this way). I'll be most interested in seeing the results.
Incidentally, re model-fitting: In the advanced version of Vensim, there's
the ability to fit a model to data by varying parameters using something
called the Powell method. I tried it with some real tracking data which I
was already using to get model parameters by a method of successive
approximations. The Vensim method did _much_ better than mine; where mine
left a prediction error of around 5% RMS, the Vensim method brought that
down to less than 2% -- with the same data, the same parameter definitions,
and the same model! Clearly, it makes a lot of difference to use an optimum
method for fitting the model.
The nearest I've come to understanding the Powell method is to realize that
it starts with an n-dimensional grid which it searches for all the minima,
and then does some kind of curve-fitting (perhaps parabolic) to locate all
the minima as exactly as possible, and then picks the best one. It runs the
model quite a few times on the way to the result -- hundreds to thousands.
I found material on the Web under "optimization."
Best,
Bill P.