Killeen; reorganization; memory;awareness; temp control

[From Bill Powers (960126.0500 MST)]

RE: incentives

     Hmmm, so the occurrence of the incentive _does_ lead to the
     establishment of a reference for striking the key, in your view.
     For a moment there I thought you were disagreeing with me. (;->

     It also leads to not pressing the key, or to pressing a key with a
     red light on over it, or to turning in a circle, or to going
     through a particular door ...


     Either you have taken the sentence beginning "Hmmm" out of context,
     or you are arguing that your proposed reorganization process
     doesn't work. I'll bet it's the former.

     Killeen's theory does NOT assume that incentive deliveries
     magically produce keypecking. In fact, he provides a detailed
     mechanism by which keypecks, once they occur, would become
     differentially "incited" by incentive deliveries, as opposed to
     such deliveries producing other activities. The ground for
     criticizing Killeen's theory lies elsewhere.

Killeen proposes that memories of responses which have occurred in
association with incentives are activated by the incentives and that
these memories of past responses somehow cause similar responses to
occur for some specific length of time proportional to the incentive. As
far as I can see, he proposes no "mechanism" at all. Everything he
proposes -- association, activation, conversion of memories into a
period of responding -- occurs by magic. He never mentions sensory
nerves, the CNS, muscles, and so on -- the actual mechanisms of

But you are right in saying that the incentive is not specific to any
particular behavior. Any behavior occurring at the time the incentive is
delivered tends to be repeated. Nothing is said that requires this to be
a feedback loop. If you pair incentives with the emission of any
specific behavior, eventually that incentive will come to "elicit" ("to
draw or bring out or forth") that behavior. Whether it does so through
the intermediary of short-term memory or in some other way is irrelevant
to the structure of the overall model. The operative part of the theory
is the effect of incentives on responses through memory associations.
Once the association is established, one incentive, regardless of its
origin, can somehow cause _a_ seconds of the kind of responding that has
occurred in association with the same incentive in the past. So I see
nothing in Killeen's theory to preclude predicting that incentives will
draw forth behavior even when they are not being caused by behavior.

Side comment:

I see a loophole, however. If you want to pair incentives with memories
of responses, what you have to do is watch for the response you want and
present an incentive when it occurs. This may seem to an experimenter
like manipulating an independent variable, but in fact the experimenter
has become part of a closed loop: from the organism's point of view, the
production of a particular behavior causes delivery of an incentive. The
pairing that is being established is really an effect of an action on
the occurrance of an incentive.

The only way to get rid of this closed loop would be to try to present
the incentive just _before_ the right behavior occurs. This would be
difficult. If the experimenter used some preparatory movement as a sign
that a behavior is about to occur, so as to slip the incentive in just
before the response, that preparatory movement would be followed by the
incentive, and might itself become the instrumental response.

So the interesting question arises as to whether it is ever the
incentive that comes before the behavior. If you just walk up to an
untrained animal and give it an incentive, it won't do anything in
particular, although it might start looking around for more of the
incentive if it hasn't enough of it. If you set up an operant
conditioning situation, the behavior always necessarily precedes the
incentive, because no incentives can occur until behavior has met the
schedule criterion. The same is true during shaping: the experimenter
has to wait for the desired behavior to show itself, and immediately
afterward delivers the incentive.

The only time when one can imagine that incentives precede behavior is
in the middle of an experimental run when incentives and responses are
already occurring in continuous alternation. But then the sequence
incentive--> behavior is simply a subjective perceptual grouping which
could equally well be seen as behavior --> incentive. There is nothing
that compels us to say that the incentives precede the behaviors.

We always come back to the same critical experiment. If incentives are
producing responses, then arbitrarily increasing the incentives should
produce more responses. If response are producing incentives in the
manner of a control system, then arbitrarily increasing the incentives
should produce fewer responses.


     If the rats are first given the opportunity to associate the sound
     of the feeder with the appearance of a food pellet in the cup,
     acquisition of lever-pressing can be quite rapid -- a matter of two
     or three pairings of lever-press and food delivery. This time-
     scale seems too short for the reorganizing system, in that
     intrinsic variables would hardly have been affected by then.

Right. I would doubt that reorganization is involved here. This sounds
more like a higher-level system that has been acquired through previous
reorganizations. We might call it a search system. What a search system
has to do is to apply some behavior to various parts of the environment
until a wanted result occurs, and then to stop the search process and
continue the behavior in that place until the controlled variable is
brought to its reference level (when possible). This is like a
systematic version of reorganization -- it is more specialized and thus
less powerful, but it is much faster.

I suppose that the main reason for starting with an association of the
sound of the feeder with delivery of a food pellet is that there is a
delay in delivery of the food pellet. The sound occurs immediately upon
pressing of the lever, but it takes a while for the food to reach the
dish. The object, I suppose, is to prevent the search from continuing
during the delay, so the wrong action seems to produce the food

It would be interesting to study this search behavior. I imagine an
array of food receptacles into which pellets can be placed silently and
without being seen. The rat has to move close enough to the receptacle
to see its bottom, and when it can see if food is there it can't see
into any other receptacle. We would expect to see the rat doing some
sort of search pattern, and stopping it whenever food was present,
eating the food and then resuming the search (as long as it was hungry).
If the pellets were placed at random, the search pattern wouldn't be
related to the pellet-appearance pattern, and would be whatever general
search pattern the rat uses in the absence of regularities. But if there
were a pattern in the appearance of pellets, how complex a pattern could
the rat learn? Could it learn to search in a simple clockwise or
counterclockwise pattern? Could it detect a repeating pattern of greater
complexity and adjust its search pattern to anticipate where the next
pellet would arrive?

This sort of experiment could easily be done with humans using a mouse
to check various locations on a screen to see if they will produce some
wanted result. We could evolve a series of patterns of increasing
complexity, and directly compare a human's and a rat's ability to match
its search pattern to a complex pattern.
     You say that reorganization involves error in _intrinsic_
     variables; this is the error found at the highest level of the
     hierarchy, correct? I have said nothing about the relationship of
     the reorganizing system to the hierarchy beyond this.

No, not at the highest level. Remember that the highest level of
controlled perception that I have proposed is called "system concepts",
a very abstract kind of perception (The United States Government, or
Physics, or Operant Conditioning). Intrinsic variables are at the lowest
level of abstraction: they are direct indications of physiological
functioning. They are even below the level of sensations. Body
temperature (in the hypothalamus) is an intrinsic variable. Circulating
glucose concentration is an intrinsic variable. Blood CO2 concentration
is an intrinsic variable. These variables may or may not have sensory
correlates; most, I presume, do not have any _direct_ sensory

The intrinsic variables and their reference levels were meant to explain
why it is that animals will learn just about anything in order to
maintain critical aspects of their bodily functions in the right states.
It's all very well to speak of incentives and rewards and
reinforcements, but what is it that gives these things any special
value? I decided that they must affect the organism in very basic ways,
and that when the basic critical variables were not in the right states,
there was a powerful motivation to get them back where they belong. And
of course, along with this motivation there had to be a means of
implementing it, which I eventually called the process of

The intrinsic variables associated with reorganization, I have always
assumed, exist at levels of functioning that are never represented
directly in experience, nor are they directly involved in the hierarchy
of control systems. The reorganizing process _creates_ the hierarchy of
control, with the eventual result of putting itself out of work. Once
the learned systems have become organized to keep the animal fed, warm,
watered, and periodically stuffed or drained, there are no longer any
large deviations of intrinsic variables from their inherited reference
levels, and the occasions for reorganization become fewer and fewer. The
random trial and error of reorganization is replaced by systematic
control processes at many levels.

The behavioral organization that is learned does not have to control
intrinsic variables directly. No behavioral control system has to sense
and control circulating blood glucose. But the hierarchy becomes
organized to control variables on which the intrinsic variables depend
-- we learn, for example, the location, seizure, and ingestion of
certain tasty objects, which we know as "food." As long as we continue
controlling for "food" intake with enough regularity, the organ-control
systems can maintain the level of circulating blood glucose near its
reference level, and there will be no need to reorganize. But we know
nothing of that; it is not part of ordinary experience. All that we need
to learn is that eating food is nice. As a side-effect, blood glucose
will remain near its reference level.
Martin Taylor 960125 16:45 --

     In my proposal, the only variable that actually induces
     reorganization (i.e. what the _separate_ reorganization system is
     controlling for) is error within the perceptual hierarchy.

Where do these error signals come from? To speak of error signals, you
have to presuppose that there is a working comparator receiving a
perceptual signal from some input function and a reference signal from
some other system.

The question I tried to answer with my version of a reorganizing system
is where the hierarchy comes from. It seems to me that your proposal
depends on the prior existence of a working hierarchy of control.

I do agree with you that some means of localizing reorganization is
essential. However, we also have to explain how an error in a
physiological variable can lead to reorganizing a behavioral system that
has only an indirect effect on that variable.
In a previous post you state that you consider all memory to consist of
neural connections (and, I presume, signals being carried through those
connections). I try to use "memory" in a much more restricted sense. "A
memory" is a reproduction of a past perception or set of perceptions
(whether accurate or not). The recording and playback of perceptual
signals is the basic idea behind my concept. I see nothing to favor any
particular recording or playback mechanism; however, considering the
vast array of things that some people remember, I doubt whether a purely
neural mechanism would suffice. I think we have to consider the
possibility of molecular storage mechanisms. Remember the subject of
Julez' experiment in which a woman was able to combine two random-dot
visual fields seen with separate eyes 24 hours apart and detect the
stereo information in them. Only an incredibly capacious memory store
could account for that.
Remi Cote 250196.1926 --

     I made the simulation you suggested (Bill P.) and there is
     something wrong. If I increase the cooling at 0.9 per dt, the
     asymptote is heading to 11. Compare to 20, it is 55%. Not 85%!

Rick Marken has already answered this. I intended for the disturbance
(the cooling) to remain constant while you look at the effect of
increasing the output gain, starting at 0.1. Try that and look at what
happens to the difference between T' and T.

A larger disturbance will always produce a larger error. If the cooling
were 1,000,000 per dt, you would expect a rather large effect!
Rick Marken (960125.1530) --
Bruce Abbott (960125.1130 EST) --

RE: awareness and control.

I know of no experimental evidence showing that awareness has any effect
on control. Remember that I distinguish between perception (existence of
a signal in a perceptual pathway) and awareness (reception of the
information in this signal by a system associated with consciousness).

It's quite clear that control does take place without awareness; all our
lower-level systems operate normally with no awareness by the whole
person of the perceptions these systems are controlling. The locus of
awareness can obviously shift, as when we attend to the spelling of a
word or to the finger movements with which we correct the spelling. But
exactly what happens to a control process when it is operating with and
without awareness remains an empirical question that nobody has yet
approached that I know of. It's likely that there is some effect -- just
watch a child holding a glass of grape juice when something utterly
fascinating appears on the TV. All the adults in the room have their
eyes on that sagging glass.
Best to all,

Bill P.

[Martin Taylor 960126 17:30]

Bill Powers (960126.0500 MST)

Martin Taylor 960125 16:45 --

    In my proposal, the only variable that actually induces
    reorganization (i.e. what the _separate_ reorganization system is
    controlling for) is error within the perceptual hierarchy.

Where do these error signals come from? To speak of error signals, you
have to presuppose that there is a working comparator receiving a
perceptual signal from some input function and a reference signal from
some other system.

The question I tried to answer with my version of a reorganizing system
is where the hierarchy comes from. It seems to me that your proposal
depends on the prior existence of a working hierarchy of control.

I do agree with you that some means of localizing reorganization is
essential. However, we also have to explain how an error in a
physiological variable can lead to reorganizing a behavioral system that
has only an indirect effect on that variable.

I think that I should probably put forward a full straw-man proposal
on reorganization, which might or might not be the same as the last time
I did so. But for now, some points.

I assume that the embryo has a control problem right from the first moment.
The cells contain control loops that sustain the stability of the various
chemical and physical components of the cell. They have to be there, or
the cell would rapidly decompose, but I'm not trying to hypothesize how
they come to exist. However, the sorts of variables that these initial
control loops control have much in common with the intrinsic variables
you hypothesize for the mature organism, and I'm assuming that the "intrinsic"
control is an evolution of the primitive control that exists in the first
cells of the embryo. I impute this initial status to your reorganization
proposal as well.

As I understand your proposal, the "intrinsic" control somehow causes to
be developed a different kind of control that we might call "perceptuo-
muscular" to illustrate that its effector reach outside the developing
organism. The perceptual control loops in this structure are segregated
from the intrinsic control loops that work entirely within the organism.
I don't understand how these perceptual control loops first come into
existence, but I'm willing to grant you that they do. My proposal has
the same hole--the effectors of the existing control systems must somehow
"learn" to reach outside the organism. I can vaguely see how that might
come about through contractions caused by changing chemistry in what
later become muscles, but it's just as much a guess as it is for your
system (as I understand it).

In my system, the chemical control systems have been reorganizing all along,
changing the "perceptions" and effectors of chemical control loops within
the cells, and at some imprecisely determined point, control has come to
include a measure of effect on the world outside the developing organism.
This means that there has also come into being some sensory apparatus
that permits the effects on the outer world to be monitored.

Reorganization is one of those magical terms you sometimes complain about.
It is something that happens, without an explanatory mechanism. In your
proposal, when some intrinsic variable is not well controlled, some new
perceptual control system may be developed that uses existing ones for
its sensory input and its effector output. In mine, when any existing
control system is in a state of sustained error, a new one may be
introduced in its neighbourhood, perhaps above it as in your system,
perhaps using the same inputs and outputs but with different PIFs and
output functions (building more of the same level) perhaps between the
existing one and the outer world. Where the new one fits depends on
where its inputs and outputs wind up being connected.

In both our systems, existing linkages can be changed, either by new ones
being made or by smooth changes in weight. The only substantive difference
is that I see the intrinsic variables as progressing continuously under
control from the first embryonic cells through to the adult organism,
with reorganization being consequent only on error, whereas you see a
novel form of control being added to the intrinsic system at some point
in development, when the organism has sufficient structure to be able
to make consistent effects in the outer world.

I carry no brief for my proposal in opposition to yours. The truth
probably lies somewhere else, perhaps with elements of either or neither.
But my proposal seems to me to be as plausible as yours (though no more so),
and worth trying at some point to simulate.

The last point you raise, about how one can reorganize a behavioural
system that has only an indirect effect on the variable that has the error,
is, I think, one of the points I was trying to address in our discussion
a few months ago about Thorndike's rat. In your proposal, what works, works
whether it is a side effect or not. In mine, the intrinsic variable sits
on top of the control hierarchy (although a vast hierarchy may have grown
up beside it to many levels higher), and control is through existing
linkages. But when a high-level system experiences sustained error, it
is changing its effectors by reorganization, and that induces transients
and therefore error at lower levels, which reorganize. The changes that
bring the high-level system back in control could be quite far from it,
and possibly not even in its direct support structure (which is very
similar to what happens in your proposal).

I've got to stop. Our modem connections now time out at 60 minutes, and
I've been on for 57.

Hope this is helpful.