[From Bill Powers (941031.1430 MST)]

RE: "Schedules" of reinforcement: Background for modeling.

If we plot the rate of behavior (B) against rate of reinforcement (R) on

a fixed-ratio (FR) schedule, we get the following:

>

> /

> /

> /

BEH B | - - - - - *

> / |

RATE | /

> / |

> /

> / |

> /

> / |

0--------------------------------------------

REINF RATE R

The slope of the line depends on the FR schedule. An "easy" schedule,

that requires few behaviors per reinforcement, would have a low slope to

the right; a hard schedule, a steep slope. If the schedule require m

presses per reinforcement, then the equation of the schedule line is

B = mR.

No matter how the animal behaves, measures of its rate of behaving and

of the corresponding rate of reinforcement will lie on the line

describing the schedule, of which the above line is an example. On the

above line there is an asterisk; this would correspond to the situation

when a certain rate of behavior B is happening, going with a certain

rate of reinforcement R. But any other point on the line would also

satisfy the conditions set by the schedule. If B = mR, then it is also

true that 10B = m*(10R), or in general kB = m*(kR).

The schedule does not determine either the rate of behaving or the rate

of responding: it merely says that whatever those rates are, the point

on the above graph that represents them will lie _somewhere_ on the

line. This line is a property of the scheduling apparatus; it is

unaffected by the way the animal behaves.

In order to predict a particular R and B pair, we need a second

relationship that describes not the apparatus, but the animal. Suppose

we say that at a certain rate of reinforcement, the animal will not

respond at all: it will just eat, as long as the reinforcements are

supplied. We can call this rate of obtained reinforcement the reference

level for reinforcement, and symbolize it as R*. R* is the value of R at

which B just becomes zero.

As the actual level of obtained reinforcement R falls below the

reference level R*, assuming the animal is acting as a control system,

the rate of behavior will rise. With a linear model, B will increase

linearly with R* - R. Plotting this on the same axis as the plot above,

we could represent this relationship as another straight line. Here is

the plot with both lines on it:

> (animal) *

> * /(schedule)

> * /

> * /

BEH B | - - - - - *

> / |*

RATE | / *

> / | *

> / *

> / | *

> / *

> / | *

0-----------------------*--------------------

REINF RATE R R*

Now there is only one point that lies on both the schedule line and the

animal line. Both the behavior rate and the reinforcement rate are now

determined.

The animal line is described by B = g*(R* - R). The constant g

determines the slope of the animal line, and R* determines its x-

intercept, which is the reference level. These two parameters, g and R*,

must be known before either the behavior rate or the reinforcement rate

can be predicted from knowing the schedule.

With both the schedule and the animal equation known, we have

B = mR and

B = g*(R* - R), from which we deduce

mr = gR - gR* or

g/m

R = ------- R*

1 + g/m

and

g/m

B = -------- mR*

1 + g/m

We can obtain the value of m from the schedule, which is known. To

estimate the animal's "error sensitivity" g, however, we need data from

at least two schedules of reinforcement:

> * #

B2 | - - - - + <-------(intersection 2

> # * /

> SKED2--># | * /<-SKED1

> # * /

BEH B1 | - - # -| - + <-- (intersection 1

> # / |*

RATE | # | *

> # / | *

> # / | * <--- animal line

> # / | *

> # / | *

> / | | *

0-----------------------*--------------------

REINF RATE R2 R1 R*

Under the assumption of a linear model of the animal, measuring B and R

on each of two schedules will provide two intersection points (shown as

+ symbols). We can draw a line through them to the reinforcement axis to

estimate the reference level of R, or R*. The model can then be tested

with other schedules to see if the B,R pair always falls on the

intersection of the schedule line with the same animal line.

In data taken by Motherall and reported in Staddon, J.E.R., _Adaptive

Behavior and Learning_, p. 214, Fig. 7.18, the stable rates of

responding and reinforcement are plotted for various FR schedules,

estimated to run between m = 1 and m = 160 in eight steps. The plot

looks like this, with an "o" showing the measured points:

> (animal) *

> *

> *

> o *

BEH B | - - - - - o

> >*

RATE | o o

> > *

> o o

> > *

> o

> > *

0-----------------------*o-------------------

REINF RATE R R*

Remember that each "o" is a point on a line from the origin extending

through the "o" with a slope representing the value of m for one

schedule of reinforcement.

The five points on the right are consistent with a linear control-system

model of the organism. As reinforcement rate becomes smaller than the

point indicated by R, however, the observed points fall below the

straight-line animal model, and actually show a decrease in behavior

with a decrease in reinforcement rate. There are several possible

explanations for the departure from a straight line.

1. A curve drawn through the o's might represent the actual error-to-

output form in the output function of the animal's control system. For

errors in the lower part of the range of error, the output rate is a

constant times the error. For errors in the upper part of the range, the

output function saturates, and then actually loses sensitivity so that

greater error produces less action.

2. Since each "o" represents a different experiment, the animal model

might still be a linear model, but the gain g might fall off as

schedules become more difficult and reinforcement rates decline.

3. On the more demanding schedules represented by the three leftmost

points, the drop in behavior rate, and consequent drop in reinforcement

rate, might be due to the animal's engaging in other activities (in

search of food) rather than pressing the bar.

4. The reference level R* might move to the left as the schedule becomes

more demanding. This entails a radical decrease in the amount of

reinforcement that the animal seeks.

Possibility 3 can be checked by inspection of the detailed records of

the experiments, computing rates only when continuous bar-pressing is

going on. Visual inspection can also establish whether the animals are

spending full time at bar-pressing on all schedules.

Possibilities 1 and 2 can be checked by having the animal switch back

and forth relatively rapidly between two schedules with only a small

difference in m between them. Presumably, if there is a general drop in

the output sensitivity but the behavior still remains linear

(possibility 2), such pairs of schedules would yield points on a line

drawn through the position of R*. On the other hand, if the plotted

points actually represent the error curve of the control system

(possibility 1), the pairs of schedules would yield points lying on the

curve connecting the "o"s above.

A check for the constancy of R* (possibility 4) can be made periodically

by switching to an FR-1 schedule and checking the asymptotic values of B

and R. If asymptote is reached quickly, the hypothesis of a constant R*

is favored.

## ···

------------------------------

Other types of schedules produce different apparatus-curves. A variable-

ratio schedule determines only that the behavior-response point will lie

in the angle between two limiting schedule lines. A fixed-interval

schedule is identical to an FR-1 schedule for behavior rates up to the

reciprocal of the fixed interval, and then is horizontal for higher

rates. A VI (constant-probability) schedule is similar, but the

transition from the 1:1 ratio to the constant level follows an average

curve of the form B = a(1-exp(-b/R)).

For all such curves, both behavior and reinforcement rates are

indeterminate without a second curve representing the organism to

intersect with the schedule curve. For the variable curves, the

intersection is a set of points lying on the organism line and bounded

by the probabilistic limits of the variable schedules. Within those

limits there is no one B-R point that can be determined, although all

will lie on the organism curve. If the organism curve also lies between

probabilistic limits, B and R can vary independently within the area of

intersection.

----------------------------------------------------------------------

Best,

Bill P.