Schedules: Background for modeling

[From Bill Powers (941031.1430 MST)]

RE: "Schedules" of reinforcement: Background for modeling.

If we plot the rate of behavior (B) against rate of reinforcement (R) on
a fixed-ratio (FR) schedule, we get the following:

          >
          > /
          > /
          > /
  BEH B | - - - - - *
          > / |
RATE | /
          > / |
          > /
          > / |
          > /
          > / |
          0--------------------------------------------
           REINF RATE R

The slope of the line depends on the FR schedule. An "easy" schedule,
that requires few behaviors per reinforcement, would have a low slope to
the right; a hard schedule, a steep slope. If the schedule require m
presses per reinforcement, then the equation of the schedule line is
B = mR.

No matter how the animal behaves, measures of its rate of behaving and
of the corresponding rate of reinforcement will lie on the line
describing the schedule, of which the above line is an example. On the
above line there is an asterisk; this would correspond to the situation
when a certain rate of behavior B is happening, going with a certain
rate of reinforcement R. But any other point on the line would also
satisfy the conditions set by the schedule. If B = mR, then it is also
true that 10B = m*(10R), or in general kB = m*(kR).

The schedule does not determine either the rate of behaving or the rate
of responding: it merely says that whatever those rates are, the point
on the above graph that represents them will lie _somewhere_ on the
line. This line is a property of the scheduling apparatus; it is
unaffected by the way the animal behaves.

In order to predict a particular R and B pair, we need a second
relationship that describes not the apparatus, but the animal. Suppose
we say that at a certain rate of reinforcement, the animal will not
respond at all: it will just eat, as long as the reinforcements are
supplied. We can call this rate of obtained reinforcement the reference
level for reinforcement, and symbolize it as R*. R* is the value of R at
which B just becomes zero.

As the actual level of obtained reinforcement R falls below the
reference level R*, assuming the animal is acting as a control system,
the rate of behavior will rise. With a linear model, B will increase
linearly with R* - R. Plotting this on the same axis as the plot above,
we could represent this relationship as another straight line. Here is
the plot with both lines on it:

          > (animal) *
          > * /(schedule)
          > * /
          > * /
  BEH B | - - - - - *
          > / |*
RATE | / *
          > / | *
          > / *
          > / | *
          > / *
          > / | *
          0-----------------------*--------------------
           REINF RATE R R*

Now there is only one point that lies on both the schedule line and the
animal line. Both the behavior rate and the reinforcement rate are now
determined.

The animal line is described by B = g*(R* - R). The constant g
determines the slope of the animal line, and R* determines its x-
intercept, which is the reference level. These two parameters, g and R*,
must be known before either the behavior rate or the reinforcement rate
can be predicted from knowing the schedule.

With both the schedule and the animal equation known, we have

B = mR and
B = g*(R* - R), from which we deduce

mr = gR - gR* or

     g/m
R = ------- R*
    1 + g/m

and

      g/m
B = -------- mR*
    1 + g/m

We can obtain the value of m from the schedule, which is known. To
estimate the animal's "error sensitivity" g, however, we need data from
at least two schedules of reinforcement:

          > * #
       B2 | - - - - + <-------(intersection 2
          > # * /
          > SKED2--># | * /<-SKED1
          > # * /
  BEH B1 | - - # -| - + <-- (intersection 1
          > # / |*
RATE | # | *
          > # / | *
          > # / | * <--- animal line
          > # / | *
          > # / | *
          > / | | *
          0-----------------------*--------------------
           REINF RATE R2 R1 R*

Under the assumption of a linear model of the animal, measuring B and R
on each of two schedules will provide two intersection points (shown as
+ symbols). We can draw a line through them to the reinforcement axis to
estimate the reference level of R, or R*. The model can then be tested
with other schedules to see if the B,R pair always falls on the
intersection of the schedule line with the same animal line.

In data taken by Motherall and reported in Staddon, J.E.R., _Adaptive
Behavior and Learning_, p. 214, Fig. 7.18, the stable rates of
responding and reinforcement are plotted for various FR schedules,
estimated to run between m = 1 and m = 160 in eight steps. The plot
looks like this, with an "o" showing the measured points:

          > (animal) *
          > *
          > *
          > o *
  BEH B | - - - - - o
          > >*
RATE | o o
          > > *
          > o o
          > > *
          > o
          > > *
          0-----------------------*o-------------------
           REINF RATE R R*

Remember that each "o" is a point on a line from the origin extending
through the "o" with a slope representing the value of m for one
schedule of reinforcement.

The five points on the right are consistent with a linear control-system
model of the organism. As reinforcement rate becomes smaller than the
point indicated by R, however, the observed points fall below the
straight-line animal model, and actually show a decrease in behavior
with a decrease in reinforcement rate. There are several possible
explanations for the departure from a straight line.

1. A curve drawn through the o's might represent the actual error-to-
output form in the output function of the animal's control system. For
errors in the lower part of the range of error, the output rate is a
constant times the error. For errors in the upper part of the range, the
output function saturates, and then actually loses sensitivity so that
greater error produces less action.

2. Since each "o" represents a different experiment, the animal model
might still be a linear model, but the gain g might fall off as
schedules become more difficult and reinforcement rates decline.

3. On the more demanding schedules represented by the three leftmost
points, the drop in behavior rate, and consequent drop in reinforcement
rate, might be due to the animal's engaging in other activities (in
search of food) rather than pressing the bar.

4. The reference level R* might move to the left as the schedule becomes
more demanding. This entails a radical decrease in the amount of
reinforcement that the animal seeks.

Possibility 3 can be checked by inspection of the detailed records of
the experiments, computing rates only when continuous bar-pressing is
going on. Visual inspection can also establish whether the animals are
spending full time at bar-pressing on all schedules.

Possibilities 1 and 2 can be checked by having the animal switch back
and forth relatively rapidly between two schedules with only a small
difference in m between them. Presumably, if there is a general drop in
the output sensitivity but the behavior still remains linear
(possibility 2), such pairs of schedules would yield points on a line
drawn through the position of R*. On the other hand, if the plotted
points actually represent the error curve of the control system
(possibility 1), the pairs of schedules would yield points lying on the
curve connecting the "o"s above.

A check for the constancy of R* (possibility 4) can be made periodically
by switching to an FR-1 schedule and checking the asymptotic values of B
and R. If asymptote is reached quickly, the hypothesis of a constant R*
is favored.

···

------------------------------
Other types of schedules produce different apparatus-curves. A variable-
ratio schedule determines only that the behavior-response point will lie
in the angle between two limiting schedule lines. A fixed-interval
schedule is identical to an FR-1 schedule for behavior rates up to the
reciprocal of the fixed interval, and then is horizontal for higher
rates. A VI (constant-probability) schedule is similar, but the
transition from the 1:1 ratio to the constant level follows an average
curve of the form B = a(1-exp(-b/R)).

For all such curves, both behavior and reinforcement rates are
indeterminate without a second curve representing the organism to
intersect with the schedule curve. For the variable curves, the
intersection is a set of points lying on the organism line and bounded
by the probabilistic limits of the variable schedules. Within those
limits there is no one B-R point that can be determined, although all
will lie on the organism curve. If the organism curve also lies between
probabilistic limits, B and R can vary independently within the area of
intersection.
----------------------------------------------------------------------
Best,

Bill P.