[From Rick Marken (961125.1600)]
Me:
The problems with your little story are so basic that it is impossible for
me to deal with them (given my current level of immaturity) with anything
other >than sarcasm;-)
Bruce Abbott (961125.1540 EST) --
I don't think you have the vaguest notion what's wrong with it (other than
that it's not e-coli reorganization), but I could be wrong. Perhaps I'm
just too thick-headed to "get it," but how about explaining for the others
who may benefit from your explication?
Well, since you put it that way -- and since there are some new folks
around -- here goes. Here is your little story (with my impertinent comments
interspersed) about how rats learn "acts" when rewarded:
Actions are movements, acts are perceived consequences of movements. An act
might be a circle, drawn in the air by the tip of my finger as a result of a
certain sequence of actions (muscle contractions)...The act is a perception
arising from behavior.
A particular act may be intended or unintended.
This is an interesting new concept -- an unintented perceived consequence of
actions. Ok. So learning (according to your model) involves changing
perceived consequences of action from being unintended to being intended.
This means developing a reference for some state of these perceptions.
When the rat first depressed the lever, that act was probably a side-effect
of activity related to controlling some other variable. When that act is
followed by the delivery of a food pellet, the rat tends to repeat the
series of acts that immediately preceded that delivery.
Interesting. So the rat has some memory of all the perceived consequences of
its actions -- intended and unintended -- that happened "prior" to the
reward. And then, for some reason, after the reward the rat "tends" to repeat
a "series of acts" that immediately preceded that delivery.
The number of perceived consequences of actions (acts) that precede (by how
much?) a reward is very large - - probably equivalent to the number of
afferent neurons in the NS. Which acts or series of acts does the rat _tend_
to repeat? Why does the rat "tend" to repeat them? How strong is the tendency
to repeat them. If a reward occurs _while_ the rat is repeating these acts,
does the rat stop repeating the acts and then restart again? If so, at what
point? How does it know about the beginning and end of "acts" and "rewards".
Your model has more loose ends than I originally suspected.
What was a side-effect of activity directed toward other goals
What does this mean? The side-effect (non-goal directed) activity was
directed toward goals all along?
now becomes the goal itself -- the rat intends to repeat that perceived
pattern of activity, i.e., what it was _doing_ on the previous occasion just
before the pellet was delivered.
OK. So the rat ends up repeating (for some unspecified reason and by some
unspecified means) one perceived consequence of its actions -- say, the
perception of the lever in a particular state. But what if putting the
level in a particular state was not the repeated act? Suppose the
rat had repeated the tongue movement perception it produced just before
the reward. How does the rat select new perceptual consequences to repeat?
Your model assumes that learning leads to the particular act or act stream
(perception of the lever in a particular state) that produces reward. But it
is often the case that quite different acts -- ones that have never before
been followed by reward -- must be produced in order to produce the reward.
The "acts" you describe are actually perceptual _variables_. Under most
circumstances, organisms can control a reward only by varying the reference
state to which the "act" perception is brought. This can be demonstrated in a
tracking task where subjects must (and DO) learn how to vary their reference
for the kinesthetic perception of handle position in order to get the reward
of having the cursor stay in the intended location.
This is basic hierarchical control theory. I suggest that you carefully
reread my "Spreadsheet hierarchy..." paper (and play with the spreadsheet
model a bit) to see how it works.
Best
Rick