Feedback functions: VI and Ratio

[From Rick Marken (950718.2100)]

Bruce Abbott (950717.1405 EST) --

On a happier note, the effect of satiation on the curve is just what is
expected under the PCT model in which the reference level for rate of
injestion is set by the higher-level system whose job is to control stomach
loading.

This is true; I ran Bill's reinforcement model with a VI rather than a
ratio feedback function. The model (which was not posted to the net,
unfortunately), comes equiped with a ratio feedback function, defined
in the statement:

ra := ba/ratio

where ra is the reinforcement rate and ba is the response rate. To create
a quick and dirty VI feedback function, I changed the above statement to:

if ba>=ratio then
   ra := ratio
else
   ra := ba/ratio
end;

Now the variable called "ratio" determines reinforcement rate in about the
way a VI schedule determines reinforcement rate, assuming that the organism
responds at regular intervals. The variable "ratio" is the maximum average
rate of reinforcment the organism can get on the schedule. If the response
rate is greater than ratio (by any amount) or equal to it then the organism
gets reinforcements at the maximum average rate at which they can occur.
(VI schedules limit the effect that response rate can have on reinforcement
rate. This is quite different than the situation with ratio schedules where
increases in response rate are directly related to increases in reinforcment
rate -- no matter how high the response rate gets). If the response rate is
less than "ratio" then I assume that the organism gets only a fraction of
the reinforcement rate possible.

When I put this VI feedback function into Bill's model (using the default
parameters used to fit the Motherall data) the result was data that looked
very much like that from interval 3 of the McSweeney VI data for subject 161
(below):

Subject 161 5-Minute Interval
Rft/hr 2nd 3rd 4th 5th
240 1728 1205 878 470
120 1207 1190 1027 643
60 922 1032 1260 914
30 713 775 636 710
15 77 89 204 165

If you lower the model's reference level you get the fall off in response
rate for the 240 (256 in the model) Rft/hr schedule but little fall off
for any of the other response rates. The qualitative fit (in terms of changing
the shape of the response rate by reinforcement rate plots for each interval)
of the model can be imporved by changing the error sensativity parameter
(of the gain change system). When error sensativity is increased, changes
in reference (from about 480 to 280) lead to changes in the response rate
by reinforcemnt rate plots that resemble the shape of these plots for
each interval in the data above.

What is interesting (to me) about this exercise is that this very simple
control of reinforcement rate model accounts (qualitatively) for the data
in two very different situations; in one situation the reinforcement is
food and the feedback function is a ratio schedule; in the other the
reinforcment is water and the feedback function is VI. The SAME model
accounted for the behavior in these two situations; the only change needed
was a change in the feedback function -- which is a property of the
environment rather than the organism component of the model.

I have a strong suspicion that the change in response rate over intervals
when the VI schedule delivers a large number of reinforcemnts (240) is not a
result of a reference level change; rather, I think it will be found that
a better model is a dynamic model that keeps the reference (for reinforcement
rate) constant; the change in response rate over trials probably reflects
the reduction of error as the perception (of reinforcement rate) is brought
closer to the reference --rather than the reference being brought closer to
the perception (as the satiety notion implies).

One reason I think that the dynamic effects are not due to reference
signal changes is that the changes in the reinforcement rate by
response rate curve with changes in reference do not really mirror
the data that well. I can get the steady state data to match the
final interval results pretty well -- but the change in reference signal
with these parameters changes the curves over intervals in a way that
just isn't quite right. Since the model matches the final interval (and
probably close to steady state) data so well, I think what's really going on
over intervals is, indeed, a dynamic change in error signal -- but this
change is probably due to the change in perception of reinforcement rate
-- it goes up-- combined with the change in gain due to the
persistance of error.

How about working on the dynamic version of the control of reinforcement
model while we're at the meeting. Bruce? I think the "plain vanilla"
dynamic version of Bill's model will account for McSweeney's data just
fine. And it will be easy to include a better implementation of the VI
feedback function in a dynamic model. The dynamic behavior of the model
might be quite different for VI and ratio schedules (I bet it is) and
then we can see if this shows up in the data.

Best

Rick

[From Bruce Abbott (950719.1310 EST)]

Rick Marken (950718.2100) --

I have a strong suspicion that the change in response rate over intervals
when the VI schedule delivers a large number of reinforcemnts (240) is not a
result of a reference level change; rather, I think it will be found that
a better model is a dynamic model that keeps the reference (for reinforcement
rate) constant; the change in response rate over trials probably reflects
the reduction of error as the perception (of reinforcement rate) is brought
closer to the reference --rather than the reference being brought closer to
the perception (as the satiety notion implies).

I'm not really in a position to comment on Bill's model as I have not had
the opportunity to see it in action yet. The problem I noted in my
description of the McSweeney VI data, if indeed a problem, would distort the
shape of the curve by making it fall off sooner with higher reinforcement
rates than would be expected if the reference level for reinforcement rate
were constant. Under the satiation hypothesis, it is not that the reference
level for reinforcement rate would be uniformly lower by, say, the third
interval, but that by that time it would be lower for the 240 rft/hr
schedule than for the 120 rft/hr schedule, which in turn would be lower than
for the 60 rft/hr schedule, etc. The effect would be to steepen the right
limb of the curve; a model assuming a constant reference would still appear
to fit the data rather well.

One reason I think that the dynamic effects are not due to reference
signal changes is that the changes in the reinforcement rate by
response rate curve with changes in reference do not really mirror
the data that well. I can get the steady state data to match the
final interval results pretty well -- but the change in reference signal
with these parameters changes the curves over intervals in a way that
just isn't quite right. Since the model matches the final interval (and
probably close to steady state) data so well, I think what's really going on
over intervals is, indeed, a dynamic change in error signal -- but this
change is probably due to the change in perception of reinforcement rate
-- it goes up-- combined with the change in gain due to the
persistance of error.

Again, I'll have to see the model at work to really see what you are
describing here, but I wouldn't expect the curves to fit a given interval
well when that fit is made using a constant reference level. What you are
calling the "final" interval" is only the last I bothered to present; as I
noted then, all 12 intervals were given by McSweeney. There is no "steady
state" reached in the sense of no further changes as a function of time
within the session; responding just continues to decline at a rate
proportional to the rate of reinforcement (i.e., rate of injestion), until
it reaches nearly zero (baseline).

As to gain changing due to a persistence of error, if this were true the
gain should be higher for the low-rate schedules than for the high-rate
schedules. Is this what you have in mind, or the opposite?

If my satiation hypothesis is correct, data from the second interval (before
significant changes in reference level have occurred) represent the clearest
picture of the relationship between reinforcement and response rates on VI
schedules under the imposed level of deprivation and reward size of this
study. Under these conditions, only the left side of the Motheral curve is
visible.

Regards,

Bruce