[From Bill Powers (941020.1030 MDT)]
Bruce Abbott (941019.1615 EST) --
Great, now we can run each others' programs. Rick, can you run mine? Can
you get Turbo Pascal 6.0 to run on your PC? Tom, have you got both
programs to run (in your spare time!)? I'm willing to stick with Pascal
for this. Dr. Sam (Saunders), what's your status in this regard? Avery
Andrews? Bill Leach? Martin Taylor? Anybody else? Who am I forgetting?
In thinking about this simulation, I would see this as the top level
control in a hierarchy, the one that gets the pigeon pecking at the
keys. I guess you'd call the error signal of this level "hunger." As
hunger increases, so does the overall rate of pecking. However, we'll
be needing something more sophisticated to properly model the
perceptual variables being controlled by the lower level(s).
There's an "even more topper" level, which sets the reference level for
food intake (r1). That's why I called it r1 instead of just r: there's
going to be an r2, etc. My philosophy of modeling is to try to find some
unit of behavioral organization that can be modeled, and then just
sketch in whatever else is needed to make it work. Once that unit is
running, more aspects of the behavior can be modeled.
I like your translations from EAB to PCT language. I agree with them.
Obviously there are lower levels of control involved: the "hunger" error
signal drives an output box that produces a rate of pecking. To position
the head and neck during pecking requires position control systems and
probably an "event" control system just above them. My implementation of
the pecking output box simply does what these lower systems would do:
makes a peck occur each time o1 reaches o1max. The error signal e1 sets
the reference level for rate of pecking. Likewise, the switching system
needs lower-level control systems that can direct a peck so it hits a
specific target; I'm sort of thinking of two control systems, one in x
and one in y, that vary the pecking location. In the wild, where the
bird is pecking the ground in a sort of random search pattern, x and y
could be directed by an E. coli program (assuming it's not a systematic
search). The pecking rate system would still be governed by the hunger-
error, independently of the position control systems. I think we can
leave the pecking process as it is -- no need to get into unnecessary
detail at this point.
There are also higher-level control systems. The amount of "hunger"
error is determined jointly by the level of the input variable q1 and by
the reference setting r1. I propose that we call the error signal the
"appetite" error, and reserve "hunger" error for the next level of
control, which will monitor a variable q2 affected by the level of q1.
q1 would represent the immediate effects of eating -- the fullness of
craw or stomach, for example -- and q2 would represent the state of
nutrition of the whole body. The decay of q1 would be an input to q2,
and q2 would have a longer-term decay constant, perhaps depending on the
rate at which work is being done. Now another level of control can be
added, which senses q2, compares it with a reference level r2, with the
error signal e2 entering an output box. The output of this box sets the
reference signal for the first system, r1. It might also set a reference
signal for the E. coli variation of pecking location, although that
part, the switching part, isn't clear in my mind yet.
With the two levels I think we can model some pretty interesting
behaviors. When the animal is maintained at a normal body weight, the
"hunger" or level-two system would experience relatively little error,
and it would set the appetite reference signal r1 at some relatively low
level. So pecking would occur at a low rate, just enough to input enough
q1 for normal bodily needs in the form of q2. But if the animal is
starved down to 80% of its free-feeding body weight, the second-level
system will experience an abnormally low level of q2, and the resulting
large error signal (hunger) will raise the r1 reference signal for
appetite. Now the first-level control system will try to maintain a
higher level of q1 by pecking faster, thus raising q2 toward its
reference level. So we should be able to reproduce some effects of the
way the animal is fed over the mid-term. Obviously the time constant of
q2 has to be measured in hours rather than minutes.
When an animal is maintained at a low body weight, there is an external
control system (the experimenter) which is adjusting available food to
keep perceived body weight (of the animal) at some externally-set
reference level (the experimenter's). This control system has far more
output capability than the animal does, and overrides the animal's
control system for maintaining its own weight. So this suggests modeling
the third level of control in the animal, body weight control. Obesity
experiments have shown that weight is very accurately and tightly
controlled (when possible). If we just took body weight as the
controlled variable, sensed in the obvious way (in the feet and
joints?), a genetically-given body weight reference signal would then
lead to a weight-error signal, which would be the basis for adjusting
the hunger reference signal r2, and so on to the first level of system
which controls for short-term food input. So while we may not be able to
model every detail, I think we can construct a good three-level model
that can be matched to a wide variety of behavior.
Right now we're using a VI-VI choice experiment. I'm going along with
this as a starting point, but I think it introduces unnecessary
complications. You will notice that in my model there is nothing
statistical; all the random variations in q1 and pecking rate come from
the fact that the schedule has a strong random component. The behavior
looks irregular because the schedule is irregular. This makes the
measurement of parameters difficult and uncertain; unless you do a huge
long experiment (during which there's no guarantee that an animal won't
change its parameters) you can't even get a good estimate of matching,
whatever you may think is being matched.
If you change the schedule (using my model) to a fixed-interval
schedule, you'll see some familiar phenomena, I think, given adjustment
of parameters. If you make the reward size large, for example, you'll
see "scalloping" in the rate of pressing; this will also happen for an
FR schedule with large enough reward size.
What I'm after is a model that will work correctly for any schedule. It
would be a real coup if we could fit a model to FR behavior over a range
of schedules, and then without changing the model at all show that it
correctly predicts behavior under VR, FI, and VI single-key schedules.
Then, of course, we could add some sort of switching control system and
see if the same basic pecking-rate model will also predict behavior
correctly in a 2-key choice situation. My hunch is that we can do this.
···
------------------------
Along those lines, the first model I sent you was intended to work with
a single-key FR schedule over a range of ratios from 1 press per access
to 160 presses per access to food (estimated from a Figure). If you will
get hold of Staddon's _Adaptive Behavior and Learning_, and look at Fig.
7.18 on p. 214, you will see two curves. These represent the equilibrium
points for a series of 8 experiments using different schedules. Each
point is the final rate of pressing and reinforcement (unfortunately
averaged over some unnamed number of animals). Looking at the lower
curve, we see that as the schedule changes toward the one-press-per-
reinforcement point, the rate of pressing declines rapidly and smoothly
toward zero, and the rate of reinforcement rises smoothly at the same
time. Extrapolating, we can see that the free-feeding reinforcement rate
(no labor required) would be about 210 "dippers". That is the reference
level: the level of input at which the output just falls to zero.
As the actual reinforcement rate falls below this level (going to the
left), the rate of responding rises to a high level, over 2000 presses
per session. This is how a control system should work: assuming that the
reference level for input is 210 dippers per session, as the actual
reinforcement rate (the input to the control system) falls below this
level, increasing the error, the output rate of responding also
increases. So the model in "varint1.pas" would work for this, too (and
does).
HOWEVER:
As you continue to the left along the curve, below a reinforcement rate
of about 150 dippers per session the response rate begins to fall below
the straight line that approximates the points on the right. In fact,
the response rate, for the remaining points to the left, turns downward
and falls rapidly with further decreases in reinforcement rate. Starting
on the left and moving right, the first three points, in fact, show the
opposite relationship of reinforcement rate to response rate: as the
reinforcement rate increases toward the reference level, the response
rate also increases.
As it happens, this relationship corresponds to the normal way of
describing reinforcement: an increment of reinforcement causes an
increment in behavior. So we can guess that this "standard" relationship
is seen only for very difficult schedules. For easy schedules, where
fewer responses are needed to obtain reinforcements, the opposite
relationship is seen: an increase in reinforcement rate goes with a
decrease in behavior rate. That relationship, which is the one expected
for a control system, works over 2/3 of the range of schedules.
In the program I sent you there is a modification that makes the
control-system model fit the entire range of this curve in Fig. 7.18.
if e[i] >= e0 then {check for nonlinear region}
g := 0.001 * k * (1000 - a * (e[i] - e0)) { if nonlinear}
else g := k; { else use linear output}
There is a threshold level of error signal, e0, so that as the error
signal rises above that level, the gain of the output function decreases
at a rate determined by the factor _a_. The gain always remains
positive, but for the left-hand point in the figure, it has declined
(for the best-fit model) to about 1/4 the value it has for the points on
the right. What this code segment does is approximate an output function
that begins to saturate for large error signals. The system behaves as
if the output/reinforcement curve had followed a straight line from the
reference point at 210 to the plotted point at the left.
What's nice about this simple change is that the model's predicted
values of behavior and reinforcement rates fall right on top of 6 of the
8 points, and the two that miss are over on the right, suggesting that
my measurements off the figure are erroneous.
The data for this figure were taken by Motherall, a student of Staddon's
in the ealry 80s. Since you know this field, could you try to get the
actual numbers from Motherall? Or Staddon? I want those last two points
to match!
There's more to Fig,. 7.18, but I've said enough for now.
It's interesting how modeling leads you toward a feasible model. In our
VIVI experiment, Rick and I have already, independently, proven that the
control system model produces anti-matching, and that if matching is to
occur it has to be done through switching control. But we have to try
all the models we can think of before picking on one.
------------------------------------------------------------------------
Best,
Bill P.