[From Bill Powers (941206.1205 MST)]
Rick Marken (941205.1230) --
Rick, this is an important point we've been leaving out; thanks for
reminding us all of it:
So a random structural change in the perceptual control hierarchy will
tend to be preserved (not reorganized away) if (as a side effect) the
change results in the reduction of intrinsic error.... Learning (from a
PCT perspective) is not a matter of "strenthening" responses that
"work" (produce a particular result).
Reinforcement theory and reorganization theory are not similar; they are
essentially opposite explanations of the phenomenon of change of
behavioral organization. Under reinforcement theory, it is the
consequence of the behavior that brings the behavior into being and
"strengthens" it. But this explanation contains some conceptual
problems. Before the behavior that is to be strengthened occurs, some
other behavior is occurring, with other consequences. Why do those
consequences not strengthen the behavior that leads to them? How does a
different consequence take over, and guide the behavior toward creating
more of itself? And exactly what effect is the consequence supposed to
have, and via what mechanism, in creating a change in the behavior?
Reorganization theory does not have these problems. Reorganization is
basically a process of random changes in organization. It is LACK of
certain beneficial effects (or presence of damaging effects) that drives
reorganization. If the organism is sufficiently hungry, or thirsty, or
hurting, or cold, it begins to alter the forms of its behavioral
systems. This process of change continues as long as the underlying
error condition persists. When a result of these changes is to
ameliorate the error condition, the interval between reorganizations is
increased; when the error condition is worsened, the interval decreases.
Only when the error condition is brought below some minimum value does
the process of change cease.
In reorganization theory there is no special effect of the consequences
of behavior on the behavior that produces them. The behavior is not
"strengthened" or made "more probable" by reorganization. Instead, the
final behavioral organization is simply the one that existed at the time
that the errors driving reorganization went to zero. If the errors had
not dropped to zero, the organization would have continued to change and
behavior would have continued to produce new and different consequences.
Grasping the difference between reinforcement theory and reorganization
theory is like a figure-ground Gestalt switch, like turning the two
faces into a vase simply by seeing them differently. Under reinforcement
theory you say that a piece of food produced by an accidental press of a
bar acts on the organism to make it press the bar again, and ultimately
again and again. Under reorganization theory, you say that as the hungry
animal roams the cage its organization is continually changing (because
of the hunger) so that it is trying to control all sorts of variables by
all sorts of means. It is not trying to get food; it is simply trying
different behaviors because of being hungry. One of the new
organizations leads it to try pressing the bar down, and the result of
this is a piece of food which decreases the error that is causing the
changes in organization. This decrease in error postpones the next
change of organization. The behavior in effect at that point will then
persist for a little while, and if it produces more food the error will
decline further and the rate of change of reorganization will become
still slower. So organizations that lead to an increase in hunger _by
any means_ lead to immediate reorganization, while those that lead to a
decrease _by any means_ postpone the next reorganization. If the
situation is favorable, the result will be fewer and fewer
reorganizations, and eventually none at all. The behavioral organization
then in effect will be the one that prevents the error that was causing
reorganization. It has to be, because any other organization would not
correct the error and thus would not prevent further reorganizations.
Note that there doesn't need to be any rational connection between the
immediate consequence of a behavior and the state of the organism. If
there is a rule that says pressing a bar to turn a green light into a
red one will result in a little jolt of intravenous food, the animal
will learn the behavior that has the consequence of turning the green
light into a red one.
This is why we were so impressed by the E. coli phenomenon: it is a
direct demonstration of the principle of reorganization. When E. coli
goes in the right direction, it does NOTHING to change the direction.
Only while it is going in the wrong direction is there any process that
alters its direction. The right direction is defined by default: it is
that direction that leads to no change in direction.
Bruce's model actually illustrated that point, more or less by accident.
The effective logical path that resulted in the correct changes in
probabilities was the one that "punished" swimming in the wrong
direction. The "pure" model, with fixed probabilities, is the basic
model of reorganization, which says that if something is wrong, tumble
right away; otherwise postpone the next tumble. So it is basically error
that gets E. coli going the right way, and keeps re-selecting the right
way every time it goes wrong. But there is nothing specific that says it
is going the right way: going the right way is simply the condition that
calls for no change. E. coli doesn't even know which way it is going.
The concept of reorganization gives us a new understanding of "reward."
It is not that getting a reward causes the behavior that leads to the
reward; it is that lack of the rewarding thing creates a state of
reorganization that will cease only when the lack is corrected.
···
-----------------------------------------------------------------------
Best,
Bill P.