[Hans Blom, 931124b]
(Bill Powers (931122.1015 MST))
Have we just had one huge misunderstanding all that time?
Very possible.
Thank you for your earnest attempt to make sense of what I say. As I have
said before, I do not want to quarrel about terminology; I want to trans-
mit some of the findings of the science that calls itself "adaptive con-
trol theory", in the hope that some of its aspects may be helpful to you,
provide you with an extra perspective on things that PCT may leave in the
dark (shadow?). In this transmission of what I have learned and found use-
ful, I try to be as objective (scientific) as I can. I have no ax to grind
FOR or AGAINST "feedforward". I just find it a useful concept, with the
emphasis on USEFUL.
There are a lot of misunderstandings in this post of yours (Bill Powers
(931122.1015 MST)), however. I want to clear up a few, and then return to
the basics. I will even provide you with a block diagram -- something I am
normally reluctant to do: formula's normally suit me much better. I will
not start a complete description of the theory. There are plenty of good
books about these themes. This is not to say that you don't understand me;
in most respects I think you do. But that impression might too forcefully
come across because in this mail I mostly concentrate on misunderstandings
and disagreements. Let me say again that I am grateful for your reactions;
they help me find out where I am unclear, either in my thoughts or in my
way of expressing them.
However, I wouldn't say "with blanked out
sensors," which overstates the case for navigating in the dark.
See below, where I discuss "perceptual input function" M.
It's become evident to me that in talking about open-loop
behavior, it's important to specify the level of organization
that has lost its feedback.
Maybe not. In my diagram below, I do not presuppose a hierarchy but
parallel (vector) processing. The basics and problems of feedforward can
be explained by this model as well, I think.
I would class the map as a perception, in the afferent path, not
the efferent path. It's a perception because we can experience it
(or we can experience it because it's a perception).
I class the map as a link between afferents and efferents. See below.
I define "control" as the stabilization of perceptions (and
often, observable external variables that they represent) against
unpredictable independent disturbances, plus the ability to bring
those perceptions to arbitrarily selected states under the same
conditions.
You forget, in my opinion, a difficult but basic control issue. Let me
quote from the book "Self-Tuning Systems" (page 2), that I referred to in
an earlier post:
"When the setpoint changes infrequently and the disturbance signal is
large, then achieving the controller design objective is said to be a
REGULATOR (italics in original) problem. Conversely, if the distur-
bance signal is negligible and the setpoint changes frequently, the
design objective is said to be a SERVO problem."
You combine both in your definition of control. Yet, in control engineer-
ing it is well-known that the two require a different approach and result
in different control laws. If, for example, you work with a PID-control-
ler, its parameters will have different settings in both cases. A good
regulator is a bad servo-system, and the other way around. Combining the
two is a difficult problem. It CAN be handled by a PID-controller, but
then its parameters must be made adaptable in some way (a popular way is
to process the setpoint signal in a "feedforward" path that adjusts the
PID-parameters). When you concentrate on and adjust for insensitivity to
disturbances, which you seem to do mostly, servo behavior -- response
speed -- will suffer. That has been my complaint, several times. Of course
you can set the PID-parameters somewhere in the middle range, but then you
will have neither a well-tuned regulator nor a well-tuned servo-system. In
my perception, organisms are usually both well-tuned regulators AND well-
tuned servo-systems. "Optimal" control systems combine the two approaches.
But that is no trivial matter.
Your definition seems to focus only on the
reproduction of an objective state of affairs, with or without
disturbances present.
Yes and no. It focusses on building up a "best" correspondence between an
outer and an inner world. It is, of course, the inner world ONLY that can
be the basis for behavior.
This difference would be unimportant if you
weren't seemingly asserting that open-loop control can be nearly
as good as closed-loop control, or that it is even the preferred
mode of action.
You misinterpret me. I say that open-loop control (I would use the term
"model-based predictive control") is the ONLY way to control if feedback
information is missing. I DO acknowledge its basic problem: jumping to
conclusions.
I think our views on what constitutes a disturbance differ
somewhat, judging from your words about Gaussian distributions.
The PCT model doesn't assume anything about disturbances except
that they be neither too large nor too rapidly-changing for the
control system to resist.
Those are ASSUMPTIONS (wired-in, "genotypic" predictions) in a PCT model.
An adaptive model would be able to ACQUIRE (be it imperfectly) "phenoty-
pic" knowledge about how large disturbances are (in the form of their
probability distribution) and how fast they vary (in the form of their
frequency spectrum).
If the system is open-loop, the
"controlled" variable will simply vary according to the amount
and direction of the disturbance.
Agreed. Yet, if there is little disturbance, "control" might be good
enough for extended periods of time.
Any disturbance can be partitioned into two components: one
component with a spectrum lying within the bandwidth of good
control (including steady disturbances -- zero frequency), and
the other containing all higher frequencies. The high-frequency
components will simply pass through to the controlled variable;
their fluctuations can't be resisted. Those effects might be
described by a Gaussian distribution.
The disturbance's amplitude distribution and its frequency distribution
are orthogonal notions. You can, for instance, have wide-bandwidth
Gaussian noise or narrow-bandwidth noise with a rectangular amplitude
distribution.
In my modeling, I consider only the low-band disturbances, and
ignore the high-band ones. I get the impression that in optimal
control modeling, the high-band disturbances are emphasized --
those that are not cancelled by the behavior of the system. Is
this more or less true?
No. High-bandwidth disturbances cannot be controlled away by ANY low-
bandwidth control system; the controller just isn't fast enough. It is the
ACCURACY of control that is the important thing. The point is that the
accuracy can be improved by collecting and using information about the
character of the disturbances.
When I say "disturbance" I mean the cause of the
disturbance: the force exerted by the crosswind, not the
fluctuations in the lateral position of the car.
I mean the same thing. See "system noise" in the diagram below. This
"cause" of fluctuations is itself unobservable except through its effects.
I think that one of our major divergences is in the way we think
about behavior in normal environments. You appear to believe that
the normal case is an environment in which regular actions almost
always can be counted upon to produce regular consequences. Only
that assumption would explain your assertion that when any
behavior is well-learned, it can be executed open-loop, with
control being required only while you're learning the actions.
That "open-loop" character shows itself in the way you walk, the way you
talk, the way you think, your habits and your character structure, in your
idiosyncracies, in the differences between us rather than in the similari-
ties, in the WAYS IN WHICH we attempt to realize our goals rather than in
the fact that both of us CAN realize explicitly stated goals.
Even small disturbances have cumulative
effects from one repetition of an action to the next, and even
during one instance of a behavior. Mathematical treatments of
control that I have seen -- particularly those which accept the
idea of open-loop control -- are mostly idealizations in which
all these little fluctuations and their cumulative effects are
ignored.
Cumulative effects do not occur in stable systems. Shaking the cup that
has a marble in it will not result in a cumulation of the distance between
marble and cup center if your shaking is not overdone. Luckily, many
aspects of the world are stable in this sense at one level or another,
even walking in a dark bedroom -- my bedroom has very solid walls...
In these treatments, all the real disturbances that can happen
are wrapped up into an innocuous term like the one you presented
as an optimal control model: a term in which it is assumed that
the average disturbance is zero, and only the distribution and
average amplitude matter.
Non-zero-average disturbances are accounted for in the model's equations.
They have (or can be given) a DC-term.
The assumption of an average value of
zero means that long-term disturbances, or even short-term
disturbances that have systematically-changing values during a
given action, are omitted from the analysis.
Not necessarily. Constant trends are easily modelled (they are constant,
after all, implying just one degree of freedom). In fact, ANYTHING can be
modelled in the internal model of the external world (although the HOW is
as yet unknown for organisms' nervous systems). The only thing that is
required is an increase in the model's complexity. There is a practical
limitation, however: the more parameters (degrees of freedom) the model
has, the more time it takes to obtain good estimates for those parameters.
Let me free-associate here a little: This is where I think model HIERAR-
CHIES come in. Given is a basic, built-in, "genotypic" model. Next, build
a simple, general-purpose model with few parameters that handles the most
important things (this assumes that we have a basic built-in sensor for
"importance", which I think we do, although our internal model of what is
important will grow over time as well). Having few parameters, it can be
constructed rapidly. Then refine the model by building higher-level (sub)-
models for the cases that the original model cannot handle well. Ad infi-
nitum or until modelling space is exhausted.
You are [in walking in the dark] controlling where you _perceive_ your-
self to be, not where you actually are.
I would agree with this statement if you broaden "perceive" to mean two
things: consultation of the feedback sensor information that IS available,
and consultation of the internal map where relevant feedback sensor infor-
mation is missing.
I think this cumulative effect of random disturbances is
overlooked in the optimal control model you posted.
In the practice of adaptive modelling you find that the DC-term of random
disturbances usually is acquired rather accurately fairly rapidly. Of
course, if it is not modeled EXACTLY, there will be drift over long time
intervals.
If there are any integrations in x(k), as there are in most real
environments, then we must consider not just d*w[k] at the
present instant, but the sum of d*w[k] over a span of previous
times. Calculating that sum would reveal the random-walk nature
of the disturbing effects, and would show immediately that those
effects can depart significantly from zero over long periods of
time.
Right. Feedforward control deteriorates as time increases.
With open-loop behavior, the controlled variable would
execute a random walk away from its initial state: there would be
no tendency for it to return to any particular state once a
deviation had occurred.
Right. But consider the time scale. The effects might cumulate very
slowly, and hence the deviations over short periods of time might be
small, even though there is no tendency to return to any particular state.
As an example, consider a planet that is being hit by debris from the
cosmos all the time, yet has a fairly stable (in the ordinary, not in the
mathematical sense) orbit around the sun.
Overlooking disturbances can easily lead to the conclusion that
the environment is stable enough for open-loop control.
Over short time intervals, such as the blink of an eye. Sometimes longer.
The movement of the planets around the sun can be predicted fairly accu-
rately over long time periods, even though there are a great many dis-
turbances. That is due not so much to the fact that these disturbances are
small, it is mostly the fact that they mostly cancel each other.
The only reason you are not blown off course by even a mild
breeze is that your direction of walking is under tight feedback
control; if the system were really operating open-loop as it
seems to be, without continuous or frequent visual feedback, you
would soon be off the path.
PCT can handle continuous visual feedback, but not FREQUENT visual feed-
back. I.e. it cannot handle missing perceptual samples. It just does not
have a mechanism to accomplish continued control when sensors are blanked.
It is that deficit that I am concerned about. I know that you have intro-
duced an "imagination connection" into your theory, but have you actually
tested it out in real applications? That is my concern.
I have worked with something very akin to that "imagination connection".
There is even a full-blooded theory about it. Here is it's diagram, sim-
plified:
-------------- organism . world
goal | | .
------>| | u .
> controller |----------------------------
-->| | | . |
> > > \|/ . \|/
> -------------- ---------- . ----------
>W^ = knowledge about W | | . | |
------------------------| ^ | . | |
> W | . | W |<-"system
------->| | . | | noise"
> > > . | |
> ---------- . ----------
>adjust | . \|/
> > . ------
--------- x | . | + |<-"obser-
> ><-------- . ------ vation
>compare> y ----- | noise"
> ><------------| M |<-----
--------- -----
.
All connections are vectors and all operations operate on vectors. The
block M is a vector "perceptual input function". It represents the fact
that generally we do not/cannot observe the full information vector of the
world, due to lack of sensors, temporarily (when you blink), or all the
time (you don't normally carry a Geiger counter). M therefore selects only
the real-world information vector that can be perceived. Generally, there-
fore, y has a (much) lower dimensionality than x. In control engineering,
M is usually taken to be a rectangular (not square) matrix with 1's on its
main diagonal, but it need not be: it can incorporate any coordinate
transformation and/or filtering.
Some aspects of the world may be unobservable (technical jargon for: there
is no method, even theoretically, to construct an internal representation
for them), others may be observable, even if we cannot directly observe
them, just like we can reconstruct an object's velocity from multiple
observations of its position. One important function of the organism is,
indeed, to create an internal "map" or model W^ of the world W. This model
contains variables (say internal representations of a real-world object's
position, velocity, acceleration) as well as "laws of nature", relations
or functions (between position, velocity and acceleration), linear or non-
linear, momentary (e.g. a square) or over time (e.g. an integral).
The comparator has access to some type of adjustment mechanism that can
"tune" the parameters of the internal model of W (the block W^). This
mechanism is, of course, internal to the organism and resembles a (multi-
dimensional) cross-correlator. Cross-correlation is something that brain
cells seem to be able to do.
Everything to the right of the dotted line represents the outside world.
The existence of an outside world is presupposed, but also that any in-
formation about it must necessarily pass through the sense organs. The
world itself cannot be known, but observations of some of its dimensions
are possible and actions can "experiment" on the world and generate in-
ternal knowledge through correlations between actions and observations.
The world W is only partly predictable. It is modelled as having a deter-
ministic/predictable part, but it also has a random part which is called
the "system noise". Moreover, some dimensions of the world W may not be
accurately observable; therefore "observation noise" is introduced in the
diagram (which can be taken as zero if you have perfect sensors for some
dimensions). The dimensionality of the information transfer through M has,
of course, a lot to do with the quality of the internal representation
that we can build of the world.
Just like the world is only partly observable, it is also only partly con-
trollable. The controller part of the system takes the information that is
available in W^, which is the knowledge it has of the world, and combines
it with the control goal(s) to compute actions u. This action is emitted
to the outside world (through muscles), as well as to the internal model
(through nerve impulses). The model's output should, of course, optimally
coincide with what can be observed in the world. But the model's para-
meters are constructed through a correlation process and this process has
limitations, as is apparent in what Skinner called "superstitious learn-
ing". Models have the same problem and there is, alas, no solution, except
through growing sensors for new dimensions of the world. In my blood pres-
sure controller, the patient's sensitivity to a drug that decreases the
pressure is estimated by correlating the drug dosage given with the pres-
sure decrease achieved by that dosage. In the initial stages of learning,
a problem may arise. If the controller starts to infuse (which lowers the
blood pressure) at the same time when the surgeon starts to cut (which
increases the blood pressure if anesthesia is imperfect), the estimation
process might erroneously conclude that the drug RAISES the blood pres-
sure. That is because the system has no information about the surgeon's
actions. We prevent this by building in and using hard-wired "genotypic"
information about the range of sensitivities that can occur in practice.
This information has been collected through analysis of the sensitivities
observed in a large group of previous patients, which is itself a learning
process that could be automated.
Note that perceptions are not directly used by the controller block; they
pass through the model and are "filtered" there. The equations show, how-
ever, that if observations are known to be noise-free, they WILL, in
effect, be used directly, because they need not be, and are not, filtered.
That's the basics. As you can see, most of the emphasis of this approach
is on acquiring an accurate internal model of the outside world, and
control is more or less a "side effect" that is determined by the goal(s)
and the internal model. Very different from PCT, but useful as well. This
is where I come from, and this is what I feel comfortable with.
Greetings,
Hans