Learning; Mirror diagram

[William_T_Powers2]

[From Bill Powers (920630.1100)]

Mark Olson (920630) --

Could someone offer a HPCT definition of learning. I know that >learning

occurs via reorganization, but that doesn't tell me what it >is. Is it a
permanent change in reference values? Is it the creation >of a new
reference signal? A new comparator? Are there different >forms of
learning? It seems that I learn HOW to do something (ride a >bike) and it
seems that I also learn WHEN to do something (when I feel >like x, I should
stay home (or go out) OR when I can't resolve >something, try not thinking
about it for a while). I wouldn't be >suprised if HOW and WHEN are
different ways of experiencing the same >thing, but they may not be.
Either way, my questions remain.

A couple of days ago I talked about "learning" as a fuzzy word with several
unrelated meanings. You can learn an address by memorizing it. You can
learn a method for solving equations by following a prodecure in a
mathematics manual. Reorganization is concerned with learning for which
there can be no rational basis and that doesn't depend just on experiencing
and remembering something new.

Learning can't result in a "permanent reference level change" because
reference levels are adjustable -- and must be -- in the HPCT model. Only
the highest reference level, for system concepts, would tend to remain the
same for long periods of time. All others change as higher-level systems
encounter errors and try to correct them. Remember that adjusting a
particular reference level has the effect of specifying the AMOUNT of a
particular perception that is to be brought about and maintained.

It would help if you were to translate from informal language, like HOW and
WHEN, into more specific controlled perceptions in terms of the model.
Knowing HOW to ride a bicycle means learning what variables to control, and
also acquiring the detailed input and output functions needed to monitor
and affect the variables in a stable way. Reorganization is required if
you've never done it before. You have to pick out many kinesthetic and
visual variables, and keep changing the way you perceive until you're
perceiving the right things in the right relationships. You have to alter
the amount of output generated by a given amount of error, and also adjust
the response to the first and second derivatives of the error, to keep from
wobbling and losing control. When I say "you" have to, I mean that
reorganization must have these effects -- you don't have much conscious
control over that process, if any. All you can do consciously is keep
trying and falling off.

Part of reorganization is an experimental shuffling of connections between
levels of control, so that a higher-level error comes to be connected to
the right lower-level reference signals. On the perceptual side, it's a
shuffling of connections from lower-level perceptions to higher-level input
functions, so you find ways of constructing higher-level perceptions out of
particular lower-level ones. And there's also the matter of the FORM of the
perceptual function -- how the higher-level perception will depend on the
lower-level signals. That's produced by reorganization, too.

Learning, real learning that occurs through reorganization, is basically a
random process. It isn't driven by what needs to be learned, but by the
consequences of NOT learning SOMETHING that works. What "works" is defined
not in terms of the perceptions and actions in the hierarchy of control,
but by intrinsic error: deviations of critical variables from their
inherited reference levels. As reorganization goes on, the form of behavior
changes; perceptions change in relation to the environment, selection of
lower-level goals changes, means of acting changes. If the changes result
in bigger or the same intrinsic error, reorganization occurs again right
away. If the change reduces intrinsic error, the rate of reorganization
slows. When intrinsic error drops below some minimum amount, reorganization
stops, and whatever organization of behavior exists at that moment has been
"learned." It doesn't matter what that organization is, as long as it
results in correction of intrinsic error.
There is no advance specification of what is to be learned; anything will
do, as long as it has the side-effect of reducing intrinsic error.

So the reorganizing system basically doesn't care about behavior,
perception, cognition, or any of those things we associate with conscious
experience. With respect to the learned hierarchy, the reorganizing system
acts like an unsympathetic boss or a cat: "I don't care how you keep me fed
-- but whatever you're doing now isn't good enough, so change it!" But this
boss, or cat, doesn't wait for you do change something. It reaches in and
twiddles something whether you want it to or not, whether you like the
result or not. And if it's still not satisfied, it does it again and again,
relentlessly, until it IS satisfied.

I assume that the reorganizing system can make arbitrary changes in any
part of the hierarchy from bottom to top. I assume that the changes are all
quite small, and that they occur at a low enough rate that the consequences
of each change have time to be reflected in the state of the intrinsic
error signal. I assume that a law of small effects is at work: small
changes have small effects. And I assume that there are some aspects of
organization that are NOT changeable: the basic kit of neurons available at
each level of control, for example, with properties that favor development
of perceptual systems and output functions peculiar to that type of
controlled variable.

Has anyone ever suggested a hierarchical reorganization system?

Yes, the thought has crossed my mind. It's possible that one level of
reorganization exists at the level of DNA; that another is involved in
development from zygote to adult; that a third is involved with the
biochemical systems (the immune systems, for example), and that a fourth is
involved primarily with the learning of motor behavior and higher levels of
neural organization.

... in my mind I visualize the reorganization system "perpendicular" to
the hierarchy, wherein the "input" is the error signal, and the >"output"

feeds into the reference signal (or perhaps the "output" of >the system
above), closing the loop. The "disturbance" is the >perceptual signal. Am
I close?

Well, not very. The reorganizing system, as I visualize it, is not
concerned with the same variables that the neural hierarchy deals with.
Think of the reorganizing system's effect not as depending on signals
flowing in the neural hierarchy, but on physical states of the organism.
Think of its actions as actually ALTERING THE WEIGHTS OF SYNAPSES AND EVEN
THE CONNECTIONS FROM ONE NEURON TO ANOTHER. The reorganizing system is not
concerned with the signals flowing in those pathways (or with their
meanings), with the possible exception of error signals (one good reason
for supposing that comparators are part of the built-in functions). It's
only concerned with the consequences of particular ways of behaving on
those variables it cares about: the ones on which life and continued
existence depend, which are largely outside the purview of the learned
systems.

If you look at the chapter on Learning in BCP you'll find some diagrams
that may help, and further discussion.

···

---------------------------------------------------------------------
Martin Taylor (20630.1010) --

Better and better. Something seems to be happening here after our long
shakedown cruise. A very coherent picture is emerging. Maybe it's just that
we have finally got past the barriers of language and have eliminated most
of the irrelevant or illusory disagreements. At any rate, I'm enjoying it.
------------------------------------------------------------------
Best,

Bill P.

[_Martin_Taylor]

[Martin Taylor 920630 15:00]
(Bill Powers to Mark Olson 920630.1100)

Part of reorganization is an experimental shuffling of connections between
levels of control, so that a higher-level error comes to be connected to
the right lower-level reference signals. On the perceptual side, it's a
shuffling of connections from lower-level perceptions to higher-level input
functions, so you find ways of constructing higher-level perceptions out of
particular lower-level ones. And there's also the matter of the FORM of the
perceptual function -- how the higher-level perception will depend on the
lower-level signals. That's produced by reorganization, too.

I'm not sure I follow this. You seem to be talking about three different
things under one label, "reorganization," and none of them are what I have
hitherto thought of as reorganization.

So let's try to iron out another wrinkle. I'll try not to use the word at
issue, atleast until after I list what I see as independent ways in which
the hierarchic control system can improve its ability to control its percepts.

First, bear with me in what seems, but is not, a digression.

An issue that's been gnawing away for a while: I take the highest level
references to be the ones you call "intrinsic." For the most part, they
provide references for perceptions based on body chemistry, I assume. You
keep talking of the System Level as the highest level that [controls?/provides
references?]. If the System Level is the highest set of ECSs, where do the
references for it come from? If the System Level simply sets reference
levels for the highest level of ECSs, then where do the settings for these
references come from? I have assumed that the intrinsic references are the
references for percepts at the System Level. The alternative seems to be
that the top level is not a control level at all, but simply a (possibly
chaotic) dynamic system.

If the intrinsic references are indeed the top-level references, then the
question of learning can be posed in two contexts: in evolution and in
individual development. In each case, the organism starts with simple
physical/chemical sensors and some way of controlling what they sense--the
bacterium wiggles faster if things are wrong than if they are right, and
thereby tends to get out of a harsh environment or stay in a benign one;
perhaps the embryo also has some way of stabilizing its internal chemistry
in a reasonable state--individual cells do. So, initially, we have a
one-level control hierarchy, into which new ECSs must be inserted by
evolution or by maturation to create new levels.

The "learning" question is how new ECSs get inserted at levels between the
top and bottom, between the intrinsic references and the sensors/effectors.
If the cleanly layered HPCT model is correct, humans have evolved so that
they can "learn" ECSs at eleven different levels of a hierarchy. Other
species may have fewer levels of ECSs, or may have ECSs with perceptual
functions of quite different kinds (though if we look at the retention of
other characteristics through evolution, it would be unlikely for us to have
ECSs of kinds very different from those of other mammals.) But a 3-level
control system would be expected to maintain control of its intrinsic percepts
(i.e. percepts corresponding to its intrinsic references) much more crudely
and over a narrower range of disturbances than would a 6-level control system.

All of the above is intended to argue that a major cause of change in a
control hierarchy (learning) is the insertion of ECSs between existing ECSs
and their input/output connections. Control becomes more subtle.

(Aside--this concept mirrors, perhaps not by accident, the notion of three-
phased learning that we proposed in 1983 for reading skills.)

(Second Aside--there is a problem here if we maintain that there are no intra-
level linkages such as "configurations of configurations." In the three-phase
learning, what is centrally learned is that large configurations can usefully
be divided into sets of related smaller ones, many of which occur in all
sorts of larger ones; the letter "c" occurs in "cat" "occur" "sick". I
take it that letters and words are both configurations. Experimentally,
there is evidence that words can be perceived both directly and through their
letters. If this is true, there is either level-jumping or intra-level
connection, neither of which is consistent with the clean layered hierarchy.
I don't want to pursue this argument further either here or in the immediate
future, but I raise a flag to mark a potential topic for later consideration.)

Back to learning. Here are some mechanisms: (1) insert an ECS between two
(or more) existing ECSs, with or without disruption of the existing links.
(Third Aside: I don't see how one can get an ECS into the hierarchy in any
other way. The new one must have a source for its reference, must get its
perceptual input from somewhere, and must send its output somewhere. All
those somewheres must be ECSs that already exist); (2) break links or make
new links between existing ECSs; (3) change the sign on an output link of
an ECS; (4) change some parameter of the perceptual input function of an ECS.
There may be other learning mechanisms, but these four seem distinctly
different, and within each there are several possibilities for further
subdivision according to the conditions that allow them to happen.

Until I read Bill's posting, I took "reorganization" to be (3), applied
in earlier discussions randomly throughout the hierarchy when the intrinsic
error is high, but as Bill agreed some time ago, reorganization must happen
in a much more modular fashion. There are too many degrees of freedom in the
system to allow global reorganization to have much probability of improving
control. But Bill seems to incorporate all of (2), (3) and (4) under the same
name. I think they should have different names, and I propose "restructuring"
for (2), "reorganization" for (3), and "adaptation" for (4).

What do these types of change in the network correspond to, in everyday
language? (1) seems to come closest to maturation--"reading readiness" and
the like, but it could include other possibilities that occur whenever we
learn to perceive a new kind of complex environmental variable (CEV). I don't
know where the new ECS would come from. There is no problem about its being
a new kind of ECS, because "kind" is in the mind of the analyst. Except for
the exact nature of the perceptual input function, all ECSs are more or less
the same, regardless of what level of abstraction they actually control.

Restructuring--type (2) learning--can happen randomly, as can reorganization
(type 3). But I would expect the occurrence to be strictly local. An ECS
that was not maintaining control despite a high output gain would be likely
to reorganize (reverse some random component(s) of its output without changing
the nature of the actions it affects) or to restructure (randomly try to do
something new). I see no rationale for either reorganization or restructuring
to affect ECSs that are happily maintaining control, or that are operating with
a low gain (insistence). There should be no need for a "reorganizing system,"
a concept Bill has mentioned from time to time, including this posting. If a
high-level ECS reorganizes or restructures, it changes the references it
supplies to lower level ECSs (though it has no information that it does so).
Possibly some of these lower-level ECSs are thereby driven out of their range
of possible control and have also to reorganize. One can generate an avalanche
of reorganization this way, if the system as a whole had been operating near
its limits; but in most cases, only one or a few ECSs are likely to be affected.

If the change reduces intrinsic error, the rate of reorganization
slows. When intrinsic error drops below some minimum amount, reorganization
stops, and whatever organization of behavior exists at that moment has been
"learned." It doesn't matter what that organization is, as long as it
results in correction of intrinsic error.

The word "intrinsic" is what I object to in this. By making intrinsic error
a stimulus and reorganization a response, you do two bad things--you bring
into existence a separate reorganization mechanism, and you do not differentiate
among parts of the hierarchy that are working badly or well. I would agree
that persistent intrinsic error should induce reorganization, but probably
only affecting the ECSs that take the intrinsic reference signals. I would
keep everything local, within the ECS: persistent error in an ECS induces
reorganization, and greater or prolonged error induces restructuring. Both
are blind.

If restructuring happens, then we must note that both the output and the
input of an ECS typically connect to the same lower-level ECS (in an efficient
hierarchy). Accordingly, not only do the actions affected by the higher-level
ECS change, but so does the CEV that it perceives. This change can be radical,
involving aspects of the world previous ignored completely.

Without restructuring or reorganization, the perceptual function of an ECS
can change, either smoothly through a process such as Hebbian learning, or
radically, as might happen if the perceptual function were programmatic
(e.g. "if input A > input B, then report 1, else report 0 as the percept"
might change to "if input A < input B, then report 1, else report 0").
Hebbian learning is guided learning. If an ECS has a percept akin to some
useful CEV, then by Hebbian learning, it may come to perceive that CEV more
precisely over time. In this case, Bill's comment:

There is no advance specification of what is to be learned; anything will
do, as long as it has the side-effect of reducing intrinsic error.

will not apply.

Another point on which I differ from Bill:

I assume that the changes are all
quite small, and that they occur at a low enough rate that the consequences
of each change have time to be reflected in the state of the intrinsic
error signal. I assume that a law of small effects is at work: small
changes have small effects.

If the hierarchy incorporates a category level, the law of small effects
will not hold. Small changes may have near zero effects most of the time,
but very large effects some of the time. And when control at different levels
occurs with different bandwidths, the rate at which the effect of any change
can be assessed will vary at least as drastically, probably more so. The
disturbances of a CEV by the world become very hard to distinguish from the
effects of a change in the sign of one small component of a complex feedback
system.

In sum: I buy the concept of random reorganization, but not that of a
reorganizing system, or the idea that reorganization depends only on intrinsic
error. And not all learning depends on reorganization (even if it incorporates
what I call restructuring). Some learning is adaptive and directed toward
regularities of the world.

Martin