[From Bill Powers (960126.0500 MST)]
RE: incentives
Bruce:
Hmmm, so the occurrence of the incentive _does_ lead to the
establishment of a reference for striking the key, in your view.
For a moment there I thought you were disagreeing with me. (;->
Me:
It also leads to not pressing the key, or to pressing a key with a
red light on over it, or to turning in a circle, or to going
through a particular door ...
Bruce:
Either you have taken the sentence beginning "Hmmm" out of context,
or you are arguing that your proposed reorganization process
doesn't work. I'll bet it's the former.
Killeen's theory does NOT assume that incentive deliveries
magically produce keypecking. In fact, he provides a detailed
mechanism by which keypecks, once they occur, would become
differentially "incited" by incentive deliveries, as opposed to
such deliveries producing other activities. The ground for
criticizing Killeen's theory lies elsewhere.
Killeen proposes that memories of responses which have occurred in
association with incentives are activated by the incentives and that
these memories of past responses somehow cause similar responses to
occur for some specific length of time proportional to the incentive. As
far as I can see, he proposes no "mechanism" at all. Everything he
proposes -- association, activation, conversion of memories into a
period of responding -- occurs by magic. He never mentions sensory
nerves, the CNS, muscles, and so on -- the actual mechanisms of
behavior.
But you are right in saying that the incentive is not specific to any
particular behavior. Any behavior occurring at the time the incentive is
delivered tends to be repeated. Nothing is said that requires this to be
a feedback loop. If you pair incentives with the emission of any
specific behavior, eventually that incentive will come to "elicit" ("to
draw or bring out or forth") that behavior. Whether it does so through
the intermediary of short-term memory or in some other way is irrelevant
to the structure of the overall model. The operative part of the theory
is the effect of incentives on responses through memory associations.
Once the association is established, one incentive, regardless of its
origin, can somehow cause _a_ seconds of the kind of responding that has
occurred in association with the same incentive in the past. So I see
nothing in Killeen's theory to preclude predicting that incentives will
draw forth behavior even when they are not being caused by behavior.
Side comment:
I see a loophole, however. If you want to pair incentives with memories
of responses, what you have to do is watch for the response you want and
present an incentive when it occurs. This may seem to an experimenter
like manipulating an independent variable, but in fact the experimenter
has become part of a closed loop: from the organism's point of view, the
production of a particular behavior causes delivery of an incentive. The
pairing that is being established is really an effect of an action on
the occurrance of an incentive.
The only way to get rid of this closed loop would be to try to present
the incentive just _before_ the right behavior occurs. This would be
difficult. If the experimenter used some preparatory movement as a sign
that a behavior is about to occur, so as to slip the incentive in just
before the response, that preparatory movement would be followed by the
incentive, and might itself become the instrumental response.
So the interesting question arises as to whether it is ever the
incentive that comes before the behavior. If you just walk up to an
untrained animal and give it an incentive, it won't do anything in
particular, although it might start looking around for more of the
incentive if it hasn't enough of it. If you set up an operant
conditioning situation, the behavior always necessarily precedes the
incentive, because no incentives can occur until behavior has met the
schedule criterion. The same is true during shaping: the experimenter
has to wait for the desired behavior to show itself, and immediately
afterward delivers the incentive.
The only time when one can imagine that incentives precede behavior is
in the middle of an experimental run when incentives and responses are
already occurring in continuous alternation. But then the sequence
incentive--> behavior is simply a subjective perceptual grouping which
could equally well be seen as behavior --> incentive. There is nothing
that compels us to say that the incentives precede the behaviors.
We always come back to the same critical experiment. If incentives are
producing responses, then arbitrarily increasing the incentives should
produce more responses. If response are producing incentives in the
manner of a control system, then arbitrarily increasing the incentives
should produce fewer responses.
···
-----------------------------------
If the rats are first given the opportunity to associate the sound
of the feeder with the appearance of a food pellet in the cup,
acquisition of lever-pressing can be quite rapid -- a matter of two
or three pairings of lever-press and food delivery. This time-
scale seems too short for the reorganizing system, in that
intrinsic variables would hardly have been affected by then.
Right. I would doubt that reorganization is involved here. This sounds
more like a higher-level system that has been acquired through previous
reorganizations. We might call it a search system. What a search system
has to do is to apply some behavior to various parts of the environment
until a wanted result occurs, and then to stop the search process and
continue the behavior in that place until the controlled variable is
brought to its reference level (when possible). This is like a
systematic version of reorganization -- it is more specialized and thus
less powerful, but it is much faster.
I suppose that the main reason for starting with an association of the
sound of the feeder with delivery of a food pellet is that there is a
delay in delivery of the food pellet. The sound occurs immediately upon
pressing of the lever, but it takes a while for the food to reach the
dish. The object, I suppose, is to prevent the search from continuing
during the delay, so the wrong action seems to produce the food
delivery.
It would be interesting to study this search behavior. I imagine an
array of food receptacles into which pellets can be placed silently and
without being seen. The rat has to move close enough to the receptacle
to see its bottom, and when it can see if food is there it can't see
into any other receptacle. We would expect to see the rat doing some
sort of search pattern, and stopping it whenever food was present,
eating the food and then resuming the search (as long as it was hungry).
If the pellets were placed at random, the search pattern wouldn't be
related to the pellet-appearance pattern, and would be whatever general
search pattern the rat uses in the absence of regularities. But if there
were a pattern in the appearance of pellets, how complex a pattern could
the rat learn? Could it learn to search in a simple clockwise or
counterclockwise pattern? Could it detect a repeating pattern of greater
complexity and adjust its search pattern to anticipate where the next
pellet would arrive?
This sort of experiment could easily be done with humans using a mouse
to check various locations on a screen to see if they will produce some
wanted result. We could evolve a series of patterns of increasing
complexity, and directly compare a human's and a rat's ability to match
its search pattern to a complex pattern.
----------------------------------------------
You say that reorganization involves error in _intrinsic_
variables; this is the error found at the highest level of the
hierarchy, correct? I have said nothing about the relationship of
the reorganizing system to the hierarchy beyond this.
No, not at the highest level. Remember that the highest level of
controlled perception that I have proposed is called "system concepts",
a very abstract kind of perception (The United States Government, or
Physics, or Operant Conditioning). Intrinsic variables are at the lowest
level of abstraction: they are direct indications of physiological
functioning. They are even below the level of sensations. Body
temperature (in the hypothalamus) is an intrinsic variable. Circulating
glucose concentration is an intrinsic variable. Blood CO2 concentration
is an intrinsic variable. These variables may or may not have sensory
correlates; most, I presume, do not have any _direct_ sensory
representation.
The intrinsic variables and their reference levels were meant to explain
why it is that animals will learn just about anything in order to
maintain critical aspects of their bodily functions in the right states.
It's all very well to speak of incentives and rewards and
reinforcements, but what is it that gives these things any special
value? I decided that they must affect the organism in very basic ways,
and that when the basic critical variables were not in the right states,
there was a powerful motivation to get them back where they belong. And
of course, along with this motivation there had to be a means of
implementing it, which I eventually called the process of
reorganization.
The intrinsic variables associated with reorganization, I have always
assumed, exist at levels of functioning that are never represented
directly in experience, nor are they directly involved in the hierarchy
of control systems. The reorganizing process _creates_ the hierarchy of
control, with the eventual result of putting itself out of work. Once
the learned systems have become organized to keep the animal fed, warm,
watered, and periodically stuffed or drained, there are no longer any
large deviations of intrinsic variables from their inherited reference
levels, and the occasions for reorganization become fewer and fewer. The
random trial and error of reorganization is replaced by systematic
control processes at many levels.
The behavioral organization that is learned does not have to control
intrinsic variables directly. No behavioral control system has to sense
and control circulating blood glucose. But the hierarchy becomes
organized to control variables on which the intrinsic variables depend
-- we learn, for example, the location, seizure, and ingestion of
certain tasty objects, which we know as "food." As long as we continue
controlling for "food" intake with enough regularity, the organ-control
systems can maintain the level of circulating blood glucose near its
reference level, and there will be no need to reorganize. But we know
nothing of that; it is not part of ordinary experience. All that we need
to learn is that eating food is nice. As a side-effect, blood glucose
will remain near its reference level.
-----------------------------------------------------------------------
Martin Taylor 960125 16:45 --
In my proposal, the only variable that actually induces
reorganization (i.e. what the _separate_ reorganization system is
controlling for) is error within the perceptual hierarchy.
Where do these error signals come from? To speak of error signals, you
have to presuppose that there is a working comparator receiving a
perceptual signal from some input function and a reference signal from
some other system.
The question I tried to answer with my version of a reorganizing system
is where the hierarchy comes from. It seems to me that your proposal
depends on the prior existence of a working hierarchy of control.
I do agree with you that some means of localizing reorganization is
essential. However, we also have to explain how an error in a
physiological variable can lead to reorganizing a behavioral system that
has only an indirect effect on that variable.
----------------------------------
In a previous post you state that you consider all memory to consist of
neural connections (and, I presume, signals being carried through those
connections). I try to use "memory" in a much more restricted sense. "A
memory" is a reproduction of a past perception or set of perceptions
(whether accurate or not). The recording and playback of perceptual
signals is the basic idea behind my concept. I see nothing to favor any
particular recording or playback mechanism; however, considering the
vast array of things that some people remember, I doubt whether a purely
neural mechanism would suffice. I think we have to consider the
possibility of molecular storage mechanisms. Remember the subject of
Julez' experiment in which a woman was able to combine two random-dot
visual fields seen with separate eyes 24 hours apart and detect the
stereo information in them. Only an incredibly capacious memory store
could account for that.
-----------------------------------------------------------------------
Remi Cote 250196.1926 --
I made the simulation you suggested (Bill P.) and there is
something wrong. If I increase the cooling at 0.9 per dt, the
asymptote is heading to 11. Compare to 20, it is 55%. Not 85%!
Rick Marken has already answered this. I intended for the disturbance
(the cooling) to remain constant while you look at the effect of
increasing the output gain, starting at 0.1. Try that and look at what
happens to the difference between T' and T.
A larger disturbance will always produce a larger error. If the cooling
were 1,000,000 per dt, you would expect a rather large effect!
-----------------------------------------------------------------------
Rick Marken (960125.1530) --
Bruce Abbott (960125.1130 EST) --
RE: awareness and control.
I know of no experimental evidence showing that awareness has any effect
on control. Remember that I distinguish between perception (existence of
a signal in a perceptual pathway) and awareness (reception of the
information in this signal by a system associated with consciousness).
It's quite clear that control does take place without awareness; all our
lower-level systems operate normally with no awareness by the whole
person of the perceptions these systems are controlling. The locus of
awareness can obviously shift, as when we attend to the spelling of a
word or to the finger movements with which we correct the spelling. But
exactly what happens to a control process when it is operating with and
without awareness remains an empirical question that nobody has yet
approached that I know of. It's likely that there is some effect -- just
watch a child holding a glass of grape juice when something utterly
fascinating appears on the TV. All the adults in the room have their
eyes on that sagging glass.
-----------------------------------------------------------------------
Best to all,
Bill P.