 ASCII.ASC follows 
[Hans Blom, 950621]
(Bill Powers (950613.1115 MDT))
Bill, thanks for your long and thoughtful reply. I am, as always,
surprised about your rapid comprehension of new stuff and the use
to which you might be able to put it. Yet some problems remain.
You're perfectly right in saying that " the outputs of your
comparators do something useful even when everything has become
stabilized." In a negative feedback control system there is
_always_ error unless for the moment all disturbances happen to
be exactly zero, and no output is required to hold the
controlled variable at its reference level.
But I am more interested in the analysis of the stable state. As
a thought experiment, set all disturbances to zero and check to
what state the system as a whole settles down to. If even when
everything has settled down, nonzero "errors" are required to
keep the system in its stable state, it is difficult for me to
think of these "errors" as errors  they can also be interpreted
as control variables that keep other variables at their correct
values. Please help me with a clearer definition of what you
consider to be error.
This is the fact about negative feedback that W. Ross Ashby got
hung up on. He saw that if the disturbance could be sensed, then
in principle his regulator could compensate _exactly_ for the
effect of the disturbance on the critical variable. Because a
negative feedback control system always had to have some error
to produce a compensating output, it could never even in
principle control exactly. Ashby decided that the only "real"
control system was the compensator.
Isn't he right IN PRINCIPLE? In your scheme of things, a negative
feedback control system always has some error (1 in 100 in your
example) to produce a compensating output, even if the error is
small. If a compensator can avoid even this small error, wouldn't
it IN PRINCIPLE be better, particularly in those cases where we
want to have a control quality that exhausts all information in
the feedback signal down to the noise level?
... The object now [in modelling a human arm] is not to achieve
perfect control and zero error, but to design a system that
shows the same amount of error as the real system [the human
arm]. This means adjusting parameters such as output amplifi
cation until the model's behavior is as close to the real arm's
behavior as possible  including imperfections of control.
I agree with the former sentence, but not with the latter. The
problem is that we do not yet have the correct model for how the
human arm operates  although some think they do  and there
fore we might have to play around with different control para
digms to find out how they might compare. Your PIDlike control
ler with its "output amplification" might give the best match,
but so might a compensatorlike controller, in which case trying
to tune an "output amplification" is the wrong approach. This
problem is far from settled, I think (and as the current litera
ture on arm models shows), so we won't know until we try out all
the possible competing approaches.
The same holds for the models we use in tracking experiments.
The model can be adjusted to track far better than the person
does.
In some circumstances, but not in others. A long standing example
of this is the approximately zero response times that subjects
soon start to demonstrate when tracking a repeating (e.g. square
wave) pattern. I have yet to see a PCTmodel that can do this
well enough, compared to a human subject. To me, this example
indicates that we ought to find out how this type of prediction
might come about.
We adjust the model's parameters to make its simulated handle
movements match those of the real person as closely as possible;
then we use the resulting parameter values as measures of the
person's control properties.
That is exactly the problem. If you presuppose a PIDlike organi
zation in the human arm when the real organization is different,
you can adjust parameters until doomsday without finding a good
enough match. This problem is the same as the problem of "unmod
elled dynamics" or system noise in Kalman Filter based control
lers, where you DO recognize the problem. Later in your mail you
do recognize the problem, though:
If you decide that you're going to build a compensator, the
constraints are very different from what they are if you decide
you're going to build a negative feedback control system. Even
the parameters of the system will be different and require
different adjustments for different reasons.
An aside:
The amplifier in the higher system [in your diagram] assures
that only a small velocity error is sufficient to change the
reference force over its entire possible range. So the velocity
error is never allowed to get very large.
A small error in your diagram equals a small "nerve current". Due
to the discrete nature of "nerve currents" (quantized as action
potentials), one would expect that small currents result in
rather noisy control signals when amplified. Given your example
of an error of 1 in 100, the error signal would be around 3
action potentials per second, assuming an upper rate of 300 a.p.
per second. This most likely would give Parkinsonlike oscillat
ions. Does this correspond to the reality of a human arm?
This [reasonable independence of elementary control systems] can
be accomplished if the perceptual signal in each system is
computed from the set of lowerlevel input signals through a
function orthogonal, or reasonably orthogonal, to the input
functions of all other control systems at the same level which
are active at the same time.
This might also be a solution to the size problem of the covari
ance matrix of the Extended Kalman Filter: it could be split up
in (almost) independent sections, resulting in far fewer varia
bles to be computed per iteration.
How do you think the orthogonalization comes about? Is it gene
tically given or learned? Could it be related to the early death
of so many brain cells in the months immediately before and after
birth?
The result is that the perceived world is partitioned into
approximately orthogonal dimensions, with independent control in
each dimension.
The Extended Kalman Filter approach could do more or less the
same, but not necessarily with independent control in EACH dimen
sion; dimensions that cannot be orthogonalized could remain
together in a vectortype control unit.
The Test for the controlled variable, as you suspect, can be
done in a simple way only when the reference level is constant.
Yes, in essence it is the scientific method: keep as many of the
independent variables (other reference levels AND other observat
ions) as you can constant, and observe the remaining relation(s)
between independent and dependent variables. Have there been
discussions about the importance of keeping perceptions limited?
I doubt that a subject who is happily tracking away and hears the
exclamation "fire!" coming from somewhere will control his joy
stick well from then on. Even "coffee!" would do it for me ;).
If there are unpredictable variations in the reference level, no
variable will prove to be perfectly stabilized by the person's
actions. However, if we could roughly identify a number of such
controlled variables, we might be able to deduce the nature of a
higherlevel system that is using them to control some higher
level variable.
Watch out. Who sets the reference level? The subject, don't we
agree on that? Doesn't that imply that there are no unpredictable
variations in the reference level?
Moreover, it is difficult to learn if you are also controlling
well. In fact, if you are controlling perfectly, you cannot learn
at all (says theory)  there is also no reason to learn (says
practice). The larger the mismatch between world and worldmodel,
the more rapidly learning takes place  if learning CAN take
place, i.e. presupposed a model with a sufficient number of
degrees of freedom.
If you want to learn (say about higherlevel systems), you might
want to consider doing this is a noncontrol context. Maybe a
stimulusresponse type of context?
In the hierarchical model, I think the whole system comes to a
state where error is more or less evenly distributed among all
the active systems. This seems to be about what you are saying.
Depends. According to the optimal control point of view, the
system stabilizes in a state where the optimality criterium
(weighted sum of squares of errors) is minimized. Differences
with your assumptions are are: The errors are weighted; some low
level errors contribute more to overallerror than others. When
there would be no noise in the system, all errors would be zero
if there are no conflicts. If there are conflicts, conflicting
goals are realized equally well (measured in terms of the error
criterium).
I think it's rather futile to speak of "optimal" or "best"
control  the whole system simply controls all the variables of
concern to it as well as it can.
"As well as it can" equals "optimal", isn't it? The problem is,
however, that you cannot measure "good", "better", "best" if you
do not have a yardstick. Hence the ubiquitous use of an optimali
ty criterium in modern control theory. According to a control
theory professor whom I recently met, he just discovered the
second control problem that could not be reformulated as an opti
malization problem. I had no time to ask for details about the
two exceptions (I'm very curious!), but this demonstrates that in
practically every case a control problem can be viewed as an
optimalization problem. In effect, doesn't The Test try to dis
cover what is optimal in a given controller?
... If the brain were born knowing everything it might be called
upon to control, perhaps it could do somewhat better.
Could do WHAT better? Which goals would be realized better, and
at the expense of which other goals? There are many more bodily
limitations, besides our brains: due to a lack of actuators, it
is generally only possible to fully realize one (conscious) goal
at the same time. More than one goal at a time, and you're bound
to find yourself in conflict (another Zen lesson: realize your
goals in succession, not simultaneously). This suggests that at
the highest level there can only be one goal.
Well, when I want to turn on the light over the sink and turn on
the disposal instead, I think of that as an "error" (blunder)
and not as optimal behavior in any sense. I just hit the wrong
switch. Of course I tell Mary that if she knew the whole story
she would see that this is really optimal behavior, not old
age....
Glad to see you understand me
Part of the environment is the other people in it. When _they_
are reorganizing, it becomes pretty difficult to predict how
that part of the environment should be modeled.
Unless you have a good model of HOW people reorganize, based on
extensive observation and experience. Some therapists seem to be
able to do just that. So it isn't impossible, just pretty diffi
cult  until you've built up an accurate model. Unless of course
you think that people reorganize in a random way. But that does
not fit in with my experience.
Note that the term "unmodelled dynamics" implies that there are
things "out there" that we cannot, will not, or do not model and
that we therefore assume to be "random". Not that part of what
happens in the world IS random (it may or may not be, we just
don't know), but that our knowledge is limited. A more extensive
model may (partially) extract knowledge (parameter values) from
that "random". In science, in particular, we should not assume
that the world is random, but that where we perceive randomness
we run into our own wall of ignorance.
... If we really had a deterministic algorithm that could
define the optimal control system  including methods of
control we haven't even thought of yet  the whole game would
be over, wouldn't it?
Yes and no. The problem is not that we cannot design algorithmic
methods to search for the optimal controller; that problem is
akin to the mechanical methods that exist to generate all theo
rems of first order logic or all possible successions in a chess
game. The problem is that WE have to specify what is optimal:
what is the goal (or what are the goals and what are their rela
tive importances); what are the constraints. It is all too easy
to forget some goal or constraint. Given a specification, a map
ping to an optimal system can in principle be found (by clever
heuristics or dumb exhaustive search), and in practice if we
specify which (countable number of) methods and how many para
meters may be used in the construction of such a system.
... the worldmodel is no longer changing x, although it may
still be changing its internal parameters a,b, and c, and thus
still changing the parameters a, b, and c involved in generating
u. Is that what you mean? The parameters are still changing and
influencing the outer control loop, I guess.
Yes, that is what I mean. Even if perception is perfect, it still
takes some time to build up the model after a change of the para
meters of the world. Think of it in this way. In a completely
noisefree system where the model structure matches the structure
of the world, the model has 4 free parameters (x, a, b, c). Now
it would take 4 observations in order to be able to solve 4 equa
tions with 4 unknowns. With system noise <> 0, 4 observations
(i.e. 4 equations) are still required to get a first estimate of
the 4 free parameters. But subsequent observations, i.e. more
than 4 equations, can be used to generate a "best fit" and thus
average out the noise, thus improving the estimates, decreasing
their variances.
What I found was that with an external disturbance of xt, the
values of the worldmodel parameters became very different from
those of the external system: _a_ became about 2*at, and _b_
about 1/2 bt.
With an external disturbance of xt, the model converges to para
meter values that accomodate the disturbance as well as possible.
So the estimated values of a, b and c will greatly depend upon
the amplitude and the type of the disturbance, but also on the
particular sequence in which the noise samples were generated.
You will find other values with other disturbances.
I use openloop systems all the time, too. My "contempt" for
openloop systems is specialized: it occurs when I see people
trying to explain behavior with an openloop system under
conditions where that is a grossly wrong model.
I can understand that. A compensator with incorrect parameters
cannot control well, if at all. That is why I advocate ADAPTATION
of the parameters of the compensator. It does seem to be a real
problem in humans, I agree, that their models frequently reach
convergence before such is warranted. Or, stated differently,
that they are not willing to revise their models when they en
counter "outliers". Freud called this "denial".
I think that as far as any ordinary organism is concerned,
transmission of genes to the next generation is strictly an
accident, not a goal. It's a sideeffect of controlling other
variables that the organism CAN sense.
What for one person is the main thing, may be a sideeffect for
another. If the two are perfectly correlated, neither The Test
nor any other method may be able to distinguish what is what.
Either we need more knowledge, or we need a religion, a set of
unprovable axioms. Ultimately the latter, I suppose.
As to the rest, you're talking about your kind of controller,
not mine. The quality of a closedloop negative feedback system
is measured by its ability to resist unpredictable disturbances,
not by the way it fails under the slightest change in its
environment.
I thought I was talking about control in general. The "ability to
resist unpredictable disturbances" can probably be expressed as a
minimization of a function such as
t = T
···

\ (x (t)  xopt (t))^2
/

t = 0
or some such (its square root, say). In that case, a rapid react
ion of x after a change of xopt will contribute significantly to
minimization of such an error criterium. I do not think that we
talk about different things!
The reason your worldmodel behaves so perfectly is that the way
you calculate u automatically takes into account any variations
in the parameters; in fact the effects of the parameters cancel
out. If the worldmodel were any more complex, so you couldn't
find the value of u just by solving an algebraic equation, you'd
have to have an automatic inversefinder as well as the system
for adjusting the forward parameters in the model.
I have no idea why you keep coming back to an "inversefinder"
time after time. My demo did not incorporate one, and neither
would a more complex model necessarily need one.
Let me explain to you when an "inversefinder" WOULD be required.
That would be in the situation where xopt (t) remains unspecified
in the formula above, because an optimal TRAJECTORY is not re
quired, but only an optimal END POINT. So the error criterium
would become something like
x (T)  xopt (T)
where xopt (T) might not only be a (vector) position but maybe
also a (vector) velocity and/or acceleration. Because xopt is not
specified for t < T, the situation is completely different: in
effect, an xopt (t) must ALSO be calculated for all t, 0 < t < T.
Because of the extra required calculation of the trajectory of
xopt (t), a solution for a "best" xopt (t) over the time from t=0
to t=T must in effect be computed. This is of course not so when
xopt (t) is PRESCRIBED from t = 0 to t = T. In that case, no
"inverse" needs to be computed.
I hope that from now on this confusion won't crop up anymore.
Where is dt, the length of physical time represented by one
iteration of the program? If the model truly had mass, then
changing the definition of dt would also require changing the
definition of at.
The time dt can be chosen at will, but must be small enough. As
usual. Taking dt twice its original value would change the
formula
x (t+1) = c + a * x (t) + b * u (t) + f (t+1)
using
x (t+2) = c + a * x (t+1) + b * u (t+1) + f (t+2)
into
x (t+2) = c + a * [c+a*x(t)+b*u(t)+f(t+1)] + b*u (t+1) + f(t+2)
= c + a * c +
a * a * x (t) +
a * b * u (t) + b * u (t+1) +
a * f (t+1) + f (t+2)
which, if we choose u (t+1) = u (t) and introduce a composite
noise g (t+2) = a * f (t+1) + f (t+2), equals
x (t+2) = [c+a*c] + [a*a] * x (t) + [a*b+b] * u (t) + g (t+2)
where you see that the original parameters a, b and c now have
different values. Does that settle your question?
When u suddenly changes, how long does it take xt to change
accordingly? One iteration, as you have designed your program.
After one iteration, xt would be changed slightly. It takes many
more iterations before xt settles to a new stable value after a
step change of u. Simplify the model equation by setting f(t)=0,
c=0 and b=1. Set a = 0.9:
x (t+1) = 0.9 * x (t) + u (t)
Assume that u has been zero for some time, and (therefore) x as
well. Now set u=1 henceforth and see how x changes:
x (t+1) = 0.9 * x (t) + 1
x (0) = 0
x (1) = 1
x (2) = 1.9
x (3) = 2.7
...
x (infinity) = 10
No physical system could respond perfectly in a single
iteration.
Certainly. But that is not what I modelled.
This surprising finding [worldmodel <> world] needs to be
explained, doesn't it? When there is a disturbance, the world
model constants come to values different from those of the real
system (I have observe this, too).
Basically there are two causes: 1) incorrect initial assumptions
and 2) learning in the face of noise is a stochastic process
whose outcome is guaranteed to be correct only ON AVERAGE. If you
choose initial assumptions with very large variances (pxx, paa,
etc) you eliminate the first cause, but at the expense of a
longer time to convergence. If you run a large number of trials
with different noise sequences, you will find that the converged
modelparameters have some probability distribution around the
worldparameter value. The width of the distribution covaries
with the noise power, all other things being equal.
Noise is a real problem in daily life as well, in humans as well
as in machines. Let me tell you an anecdote from my initial
trials with an adaptive blood pressure control system, in which
the effect of a hypotensive drug (sodium nitroprusside, SNP) on a
particular patient was to be estimated by correlating the change
in the blood pressure with the infusion flow administered to the
patient, in a way similar to what happens in my demo. This esti
mate was required because the patient's sensitivity to the drug
varies so much (a factor of 80), that a nonadaptive controller
cannot perform well enough.
Now sodium nitroprusside is a drug that lowers the blood pres
sure. Yet, when the controller started to infuse SNP, the blood
pressure went rapidly and significantly UP. So the model decided
that the bparameter was positive, i.e. that SNP _raises_ the
blood pressure rather than lowering it. So the controller de
creased the SNP flow rate. Then the pressure went down. But too
far, below the setpoint. So more SNP was infused. The blood now
pressure went up again. By now, the model was pretty certain that
SNP is a drug that RAISES the blood pressure.
Unknown to the model was the fact, that at the time when the
infusion started, the surgeon made his first cut; and also, that
pain suppression was inadequate. So what made the blood pressure
go up was the pain due to the surgeon's cut, which had a much
more powerful effect than the (still small) SNP flow. After the
cut was over, the blood pressure dropped, simultaneously with the
decrease in SNP flow rate.
Well, you get the picture. A sudden noise burst at the wrong
moment may be explained as significant. And when it repeats, as
very significant. This leads to superstition. There are solutions
to the problem of building up incorrect knowledge, of course, but
these require more sensors, a more complex worldmodel, and/or
builtin limits on how far the values of the parameters can vary.
We chose a combination of the latter two approaches.
Greetings,
Hans