[From Bill Powers (950623.1410 MDT)]
Bruce Abbott (950623.1135 EST) --
It is easy to modify the model, however, so that it no longer tends
to infinity. For example, think of reinforcement as being
equivalent to charging a capacitor. In this case the increment in
response strength is proportional to the difference between its
present strength and some maximum possible strength.
Now you have to define "response strength," and give it special
properties that just compensate for the instability. Better model this
so I know what you're talking about. I agree that once such a problem is
noticed, you can always tack on some ad-hoc feature to the model for the
sole purpose of getting rid of that specific problem.
The discriminative stimulus should not be viewed as a stimulus that
elicits a response. Pr(B|S) is not the probability that S will
lead to B but the probability density over time of B, given that S
is present. If this probability is high and S is short, the result
may look like S-R: S occurs and R follows almost immediately. But
this is only an extreme case.
... the use of probabilities is not "dictated" at all, as one may
work directly with the rate measure.
In that case you have a problem unless you add some ad-hoc feature to
the model to prevent it from running away. "Running away" does not mean
going to infinity; it means that there is no stable state between the
physical limits of the variable.
In cases where what varies is the magnitude of the response (i.e.,
response force), the reinforcement principle has been applied
successfully without recourse to a probability interpretation.
How? Was this done without introducing any artificial limit or
nonlinearity to prevent runaway?
This is why reinforcement theorists talk about reinforcement
affecting the "strength" of behavior rather than its probability.
But isn't response strength measured in terms of resistance to
extinction? That's different from saying that a response to a stimulus
exerts a force of 1 gram or 10 grams on a key. How would you put this
response strength into a physical model?
Rate or probability are considered to be measures of response
strength, but so would the force exerted on the lever if the
reinforcement contingency required a specific force before delivery
of the reinforcer.
What would you call it if the force exerted on the lever continuously
affected the amount of reinforcement delivered? Would you also have a
reinforcement strength in that case? What's the physical correlate of
"strength?"
As each response continues to produce the "same" reinforcement, the
rate of behavior will increase to some value dependent on the value
of the reinforcer to the subject and the cost of responding (e.g.,
effort).
OK, so you have to make the behavior rate depend on the value of the
reinforcer to the subject, minus the cost of responding. How do you
introduce the value of the reinforcer to the subject in the model? Does
this require adding something to the model equivalent to a reference
signal?
The "amount" of reinforcement conceived as reinforcement per
response is constant in this scenario and this relationship
determines how much response "strength" is incremented with each
successive reinforcement, up to its limit.
If this incrementing is linear, then you still have runaway if the loop
gain is greater than 1. There is still no stable state between 0 and the
limit. You have to introduce just the right nonlinearity to get the
response rate and reinforcement rate to level off at a finite value
between the limits.
This "molecular" analysis based on the immediate consequence of
each response can be supplemented with a "molar" approach based on
rates if one takes the view that the subjects in these experiments
can integrate events over time. In this view, receiving individual
reinforcements at a higher rate is more reinforcing than receiving
them at a lower rate. A brief increase in response rate (e.g.,
accidental lever presses as byproducts of general activity) result
in an increase in reinforcement rate (increased rates of lever-
pressing are reinforced by increased rates of food pellet
delivery). Decreased rates of lever-pressing lead to decreased
rates of food pellet delivery (punishment). The molar approach
thus predicts that response rates will tend to increase toward the
maximum sustainable rate as determined by reinforcer value and
response cost.
Now we're back to the imaginary case in which an increase in
reinforcement rate goes with an increase in behavior rate. In general,
what actually happens in most cases is the reverse of what you describe
above. If an animal is feeding itself on a FR1 schedule and it
accidentally hits the bar to produce extra reinforcement, it will slow
its behavior to return to the previous average relationship between
behavior rate and reinforcement rate (on a fixed schedule).
This is what is observed, and not the bi-stable behavior you
predict on the basis of the positive feedback relationship as you
conceive it. The analysis you propose would lead to instability of
behavior on a ratio schedule: any decrease in response rate
(perhaps resulting from the rat's momentary need to scratch an
itch) would be expected to lead to total extinction of the
response. It doesn't happen, and it isn't predicted to happen by
reinforcement theory.
You may have misread my post. I said that the instability will occur
only if the loop gain (k1 * k2) is greater than 1. If it is less than 1,
the effects will be amplified but there will be no runaway. When no
runaway is observed, the loop gain must be less than 1.
To avoid this obviously inappropriate result, the meaning of "more"
had to be modified so that the outcome was not a runaway condition.
But this doesn't solve the problem, does it? Your analysis still
has the instability problem I pointed out whether "more" is an
increase in rate or an increase in probability.
If the loop gain is less than 1, no instability problem exists.
···
----------------------------------
Let's apply this model to the steady-state situation involved in
performance on a simple CRF (1 response per reinforcement) schedule
as ordinarly studied in the operant chamber. A hungry rat is
placed in the chamber. We might conceive of the set point for rate
of food pellet consumption as, say, 30 pellets per minute (30 ppm)
under these conditions (essentially continuous eating). However,
the apparatus limits the maximum rate to 10 ppm because of the
delays involved in depressing the lever, moving to the food cup,
picking up the food, devouring it, and returning to the lever.
Thus, there is NO WAY that the rat can reach its reference level
for this quantity, although it can reduce the error by a
significant amount through lever-pressing (and by no other means).
I believe that control theory predicts that the rat (once it has
learned what to do) will develop a rate of responding that
minimizes the error, up to the point where the error is reduced
enough to bring the output below maximum. Depending on the gain,
there will be a region within which the rate of responding will be
a direct function of the magnitude of the error.
All right, you're setting up a special situation in which the rat can't
reach its reference level for nutritive input because there are physical
limitations, combined with the reward size, that prevent its doing so.
The exact prediction that a control model would make would depend on the
loop gain of the rat (measured without these restrictions), on the
measured reference level for food intake, and on the degree of the
restrictions. If the rat had a high loop gain, we would expect the
behavior to remain at the maximum rate under all conditions.
Let's graph the situation:
450*
*
* x
*
* x
300* - - - - - +
BEHAVIOR * | x
* |
* | x
RATE * | ref level
150* | x <--error--> |
* | |
* | x |
* | |
* | x |
*************|*************|************|**********
10 20 30
REINFORCEMENT RATE
The line of x's represents the minimum consistent output sensitivity of
the control system in units of presses per minute per reinforcement per
minute of error. To fit your conditions it has to be about 30
(reasonable).
If we now increase the ratio requirement, the maximum rate of
reinforcement available on the schedule is less (say, 5 ppm).
It would have been easier if you had specified the new ratio requirement
and then figured out the result, but we can still get there. If the new
rate of reinforcement is now 5 reinforcements per minute, we can deduce
that the new ratio is 390/5 = 78.
450*
*
390* - - x
* |
* | x
300* |
BEHAVIOR * | x
* |
* | x
RATE * | ref level
150* | x <--error--> |
* | |
* | x |
* | |
* | x |
******|********************|************|**********
5 10 20 30
REINFORCEMENT RATE
This means that a larger error remains after responding has reached
its maximum rate than was the case with the CRF schedule. If
response rate on the CRF schedule was already at maximum, it will
still be at maximum on the new schedule; if not, then it will
increase.
Right.
But this analysis ignores response cost. Assume there is a second
control system for "response effort" and that the reference for
this perception is set to a low value. We now have two control
systems in conflict when the rate-of-eating system brings the rate
of lever-pressing up and thereby brings response effort above its
set point. The result is that neither system can reach set point
and rate of responding stabilizes at a kind of equilibrium value
between the two set points.
Actually, if you model this (as I have done), you will find that the
only effect is a lowering of the effective responses-per-unit-error
curve's slope. The slope does not reverse; it doesn't even become
horizontal even if you make the cost very high. I was doing this in an
effort to get the line of x-s to develop a curvature and go down again
as the ratio requirement increased, to fit the Staddon/Motherall data.
It doesn't even help to make the cost vary as the square of the response
rate; the curve still never bends downward.
The only way to get the curve to bend downward is to use a cost-benefit
control system as you suggest, but set its reference level so it doesn't
come into effect until the error has reached about 60% of the maximum
possible error. Then by adjusting the gain of this loop you can get the
whole curve, over ratio requirements from 1 to 160, to match the real
data very closely.
You don't see the negative feedback regulation of food rate in the
control model because food rate never reaches its reference value.
For this reason I feel that your control model of the right portion
of the Motheral curve does not apply in the way it would if control
could be achieved.
As the above curves show, there is negative feedback regulation
throughout. However, your example used unrealistic numbers. In the S/M
data, the responses-per-unit-error slope was about 20. Increasing the
ratio requirement from 1 to 2 decreased the reinforcement rate by about
5% (from 200 to 190) and doubled the response rate (from 200 to 380),
roughly. Since the maximum observed response rate was about 3000, there
is no question of being near or above the maximum response rate limit.
I would prefer that in giving examples, you would use real data. Also,
when you assume the effect of some added feature of the system, it would
be a good idea to set up the model and run it, to see if your intuitive
prediction of what would happen actually happens.
As I hope I've communicated above, the reinforcement model can
handle the data without restricting its application to the lower
end of Motheral's curve. However, the application at this end does
partly involve considering the selective effect of reinforcement
when alternative sources of reinforcement are available for
different behaviors, as you suggest.
I think you need to actually set up the reinforcement model and run it
under conditions of various ratio requirements, with and without the
"alternative behaviors" provision (which applies, as you say, to the
left side of the figure). In your above example, you assumed that the
reversal of the relationship on the right side was due to a limitation
on the maximum response speed of the organism; as a quick look at the
S/M data will show you, that was not a consideration. I need to see a
reinforcement model that will behave as in the right side of the S/M
data as the ratio requirement is changed in the range from 1 to about
40. I already know how a control-system model will behave; for
comparison, we need the behavior of a reinforcement model.
This is a nice resolution, and essentially agrees with the analysis
I presented in pure reinforcement terms. Clearly the upper limb of
Motheral's curve is better explained by considering the way the
mechanism of the control model would be expected to operate under
the conditions of the study than by the complex analysis that must
be developed under reinforcement theory (although one CAN be
constructed). And as you suggest, the lower limb can be understood
at least partly by considering the selective property of the
reinforcement principle. An interesting project would be to
consider how such a selective effect might be handled by PCT.
Good. I suggest that we make this analysis an explicit goal of the rat
experiments that are just about to be done. As to the PCT analysis of
the left-side data, I already have an ad-hoc PCT model of that, but
let's not worry about that for now. In the paper, we can simply point
out this area of comparison as being left for future study.
If a reinforcement model CAN be made to fit the right side, I suggest
strongly that you go ahead and develop it, however complicated it gets.
The very complication, in comparison with the simplicity of the control
model, may carry a message of its own. And if it should prove that you
can't find any reinforcement model that, under the observed conditions
and observed range of possible behavior rates, will fit the data, that
will carry an even more important message.
-----------------------------------------------------------------------
Scott Brandon (950623) --
Welcome aboard! I hope that our sociologists will chime in and help you
get some threads in that area started.
-----------------------------------------------------------------------
Best to all
Bill P.