[From Bruce Abbott (960106.1400 EST)]
Peter Killeen (1994) proposed what he terms a "mechanics of behavior" and
subsequently discussed some aspects of this approach in a 1995 JEAB article
that was a recent topic of discussion here on CSGNET. In this post I
undertake a review and partial analysis of Killeen's proposal as presented
in that 1995 article. (I have not yet received a copy of the 1994
treatment, which is more extensive.)
Killeen's "mechanics of behavior" is based on three principles:
(1) Incentives excite responding.
(2) There is only so much time available in which to respond.
(3) For a particular target response to be differentially excited by an
incentive, it must be paired with that incentive; they must coreside
in the animal's short-term memory.
The 1995 paper focuses mainly on Killeen's first principle, which he
expresses mathematically as follows:
(1) A = a*R, where A = arousal level,
a = specific activation
R = rate of incitement
Arousal level is defined as "the amount of responding elicited by a schedule
of incentives in the absence of competition from other responses." Specific
activation is the number of seconds of behavior incited by one incentive;
experimentally it was determined that, under constant conditions, one
incentive would generate a total of "a" seconds of behavior. The rate of
incitement is the the rate at which the incentives (reinforcements) are
delivered, in reinforcers/second. At rate R rft/sec, the incentives will
incite a*R = A seconds of responding per second.
For this analysis Killeen uses a simple method to translate A into B, the
rate of behavior:
(2) B = A/s,
where s (delta in Killeen's paper; my keyboard doesn't seem to have a delta)
is the minimum inter-response time in seconds. s converts A seconds of
responding per second into the rate of discrete responses under the
assumption that each s seconds of responding is equivalent to one response.
Because s is a constant, in most of his derivations Killeen sets it to 1.0
and thus writes equation 1 as
(3) B = a*R
This is Killeen's "first principle" of behavioral mechanics. Basically, it
says that each delivery of a reinforcer "incites" "a" seconds of activity,
which through learning (not treated here) gets translated into the
instrumental behavior, here represented as discrete responses (e.g., lever
pressing). An analogy: This is like a gasoline engine whose fuel pump
delivers gasoline into the engine in spurts, each spurt producing so many
revolutions of the crankshaft. If the pump delivers another spurt before
the engine dies, off it goes again for another set of revolutions.
On a fixed ratio schedule, each n behaviors produces one incentive. In my
analogy, the fuel pump is being driven by the engine at n revolutions per
squirt. The environment function for a ratio schedule is
(4) R = B/n, where n is the ratio of responses to incentives.
To apply Killeen's first principle, one must determine the specific
activation, which depends on both the incentive value of the incentive and
on the current level of hunger. If a = the specific activation, then
(5) a = v*h, where v = incentive value in response-seconds, and
h = level of hunger.
v determines how many seconds of activity will be produced by an incentive
under a given level of hunger and is determined empirically. The level of
hunger is a function of the level of deprivation as follows:
(6) h = Y*d, where Y (gamma in Killeen's paper) is just a constant of
proportionality.
Y converts units of deprivation into units of hunger. Killeen notes that
simple proportionality is probably too simple (the actual function is
probably nonlinear) but adopts the linear model here for simplicity of
treatment in the present exposition.
d in this model is viewed as the amount of food in grams below stomach
capacity; Killeen notes that this is a simplification compared to the
variables probably actually contributing to hunger, but adopts this
definition of deprivation for simplicity. Killeen computes d as follows:
(7) d = d0 + (M - m*R)*t,
where d0 is the initial level of deprivation in grams of food, M is the rate
of depletion in grams/sec, m is the magnitude of reinforcement in grams, R
is the rate of reinforcement in rft/sec, and t is the elapsed time since the
beginning of the session. But this equation assumes a constant rate of
reinforcement, R, over time t. An appropriate equation for simulation would be:
(7a) d = d + (M - m*R) * dt, where dt is the time per iteration.
At the start of the simulation, d would be set to d0.
In my control-system simulation, I used F (food amount in grams) as the
controlled perception and the following equation to determine F on each
iteration:
(8) F = F + (m*R - M) * dt
But the level of deprivation in Killeen's model is just the difference
between a full stomach and the current level:
(9) d = Fr - F, where Fr = the food reference level (full stomach), and
F = the current level.
Substituting equation 9 into equation 7a:
Fr - F = (Fr - F) + (M - m*R) * dt;
-F = Fr - F - Fr + (M - m*R) * dt;
F = F - (M - m*R) * dt;
F = F + (m*R - M) * dt,
which is just equation 8. Killeen's equation is thus equivalent to mine,
once his equation has been properly written for the simulation.
Equation 6 can be treated similarly. Substituting (Fr - F) for d gives
h = Y*d = Y*(Fr - F)
But (Fr - F) is just the error in the control system model. Thus
(10) h = Y*e, where e = error in grams.
Thus hunger (h) is proportional to error, as seems reasonable.
At this point it would be instructive to compare the simulation equations
for the control model to Killeen's equations. The table below gives the
control model equations, Killeen's equations, and Killeen's equations
restated in terms of the control-model variables.
Control model Killeen's model Equivalent formula
···
-----------------------------------------------------------------------
(1) F = F + (m*R - M)*dt d = d + (M - m*R)*dt F = F + (m*R - M)*dt
(2) p = F p = F
(3) e = Fr - p e = Fr - p
(4) h = Y*d h = Y*e
(5) g = constant a = v*h g = v*h/s
(6) B = g*e B = a*R/s B = g*R
(7) R = B/n R = B/n R = B/n
-----------------------------------------------------
Using the equivalent formulas for Killeen's equations, lines 1, 2, 3, and 7
are identical in the two models. The only significant differences between
the two models are found in lines 5 and 6, and these make ALL the
difference. The control model translates error directly into a rate of
behavior through the output gain parameter, g. Killeen's model uses error
to produce a proportional hunger, which determines (together with the value
parameter v) the rate of behavior "incited" by the current rate of
reinforcement. On line 5 of the equivalent formulas I translate these
effects and the divisor "s" as "g," the output gain. This output gain then
determines the rate of behavior from the rate of incitement in line 6.
Here is a block diagram of Killeen's model:
Fr
>
v d h
+ --->[X]------>[h=Y*e]----------+
p | |
v v*R v
[input f] +------>[X]---->[output f] B = v*h*R/s
^ | ^ |
> > > >
> >v | R | B
> > > >
> > > v
+----[X]<------[m*R]<-----+----[environment f]
F ^ m*R R R = B/n
>
M
For clarity I have used F and Fr in the diagram, which appear only
implicitly in Killeen's original equations. The diagram reveals that
Killeen's model contains two loops. The main loop is essentially my
food-amount control system, although with a different output function. The
inner loop is Killeen's reinforcement mechanism. Output (B) produces
incentives at rate R through the environment function; These incentives have
magnitude m (in grams), which contributes to the value of the incentive, v.
Incentive value and rate combine multiplicatively with hunger (h) to set the
gain of the inner loop.
What happens in that inner loop can be seen by examining lines 6 and 7 in
the system equations:
B = a*R/s
R = B/n
Substituting B for R in the first equation gives
a
B = B*---
n*s
But a = v*h (line 4), so
v*h
B = B*---
n*s
Call the ratio on the right side k, the loop gain. At each succeeding time
period dt, B will be multiplied by k. What happens for different values of
k is given below:
k > 1 B ---> infinity
k = 1 B ---> B
k < 1 B ---> 0
B will not actually reach infinity because the interresponse time cannot
fall below its minimum, s, but it will quickly reach the maximum rate of 1/s.
The reinforcement loop is a positive feedback loop in which B increases
exponentially for k > 1. In the paper, Killeen mentions that a more
complete version of his model would have to allow for the "blocking" effect
of previous incentives: The incitement produced by a given contingent
incentive tends to be reduced if another incentive has been delivered
recently; this blocking effect declines with time since previous incentive
delivery. The effect of this would be to reduce the loop gain of the
reinforcement mechanism as incentive rate increases; with the right
parameters this effect might prevent the runaway to 1/s responses/sec, but I
don't have Killeen's more detailed paper yet and therefore can't say how he
implements this.
I have presented Killeen's equations as he gives them to emphasize the
similarities in structure of the two models, but the Killeen equations as
given above will not produce a workable simulation. The equations in lines
6 and 7 must be modified before they will work in the iterative case.
Consider line 6:
B = a*R/s
The parameter "a" is the number of seconds of responding incited by one
reinforcer; multiplying this by R, the rate of reinforcement, gives the
number of seconds of responding elicited per second of session time. But
each interation will consume only dt seconds, where dt is some small value
(0.1 sec in Bill's program). On a single iteration only dt seconds of
incited responding could be "consumed", which imposes a maximum rate on
behavior of dt/s responses per second. The simulation needs to take this
into account by dealing directly, not with the _rates_ R and B, but with the
_amounts. If the rate of reinforcement is R responses/sec, then there will
be R*dt reinforcers in dt seconds. The _number_ of responses in dt seconds,
b, will then be
b = a*r/s, where r = the _number_ of reinforcers in dt seconds.
b can never be greater than dt/s, so b should be limited to this value in
the simulation.
In the Killeen equation in line 1, r can be substituted for R as follows:
d = d + (M*dt - m*r)
The rates can be computed from r and b by dividing each by dt:
B = b/dt
R = r/dt
I have implemented this model by modifying Bill's FOOD1.PAS and will provide
it in another post, after I have had time to clean it up. At the initial
deprivation level (10 grams) responding quickly goes to its upper limit and
stays there until the level of deprivation drops sufficiently to bring the
reinforcement loop gain below 1.0.
Killeen develops his model first for interval schedules and only later applies
his "mechanics" to the ratio case. For interval schedules, rate of
reinforcement changes rather little with response rates beyond a certain
minimum response rate, in which case m*R would be nearly constant. For high
enough rates, Killeen's mathematical derivations from his model, which
incorrectly assume that m*R is constant over elapsed time t, probably give
results fairly close to the true ones, although I need to check this out via
simulation. For the ratio case, however, and for the interval case with low
response rates (which then essentially reduces to the ratio case), his
incorrect derivations deviate strongly from the actual behavior of the system.
Some Conclusions
1. Killeen's "mechanics of behavior" approach represents a step forward
compared to the usual EAB functional analyses. Although terms like
"deprivation" and "hunger" seem to represent the usual vague
psychological entities, they are for the most part just names for
physical variables, which can be at least potentially observed and
measured. These variables are linked together in a particular model
in the same way that our control-system models are constructed.
2. The model for ratio responding Killeen develops contains a control
system for the level of food in the gut, although its output mechanism
is a bit strange.
3. The output mechanism in Killeen's model embodies positive feedback, and
would have behavior rate running away to its max rate or to zero, depending
on the output gain, which is proportional to the error in the food-level
system and the response-eliciting "value" of the incentive, and inversely
proportional to the ratio requirement. Given that v and n are constant,
this machine would respond at max rate until enough food had been
consumed to bring the error in the food-amount system (and thus h) low
enough to reduce the output gain below 1, at which point responding would
rapidly cease.
4. Killeen's mathematical derivations for average rates of behavior and
reinforcement over elapsed time t are incorrect as they assume the rate
of reinforcement to be constant over that time, and they generally are not.
Clearly, Killeen is trying to move EAB in the right direction, toward a
mechanistic analysis of behaving systems and away from a purely descriptive
analysis. In his 1995 paper, he tries to show how this approach can be used
to account mechanistically for the relationships only described in such
approaches as "behavioral economics," which deal only in externally
observable variables such as costs (responses or effort required) and
benefits (magnitudes of rewards).
Unfortunately, his mathematics is not up to the job, nor does he seem to be
aware that his model could be submitted to computer simulation as a way to
check on his derivations. He probably does not know that his model includes
a control system for food-amount, and he probably does not know how to
properly analyze such a system, although he manages to create one
nevertheless by following the intuitive logic that adding food will reduce
deprivation (error), and that reduced error will lead to reduced behavioral
output.
Killeen's reinforcement mechanism works just the way Bill Powers suggested
for reinforcement in general, based on the notion that each reinforcer
delivery will increase the rate of behavior, thus increasing reinforcer
deliveries, i.e., positive feedback. It does not work the way Killeen
thinks it does, based on his faulty derivations; it does not produce rates
of behavior commensurate with the level of deprivation.
Killeen's analysis sticks with the assumption that each incentive delivery
produces a fixed amount of behavioral "arousal" in conjunction with the
current level of hunger (error), which serves to produce a fixed number of
responses and ultimately, through the schedule function, another incentive.
The arousal must be sufficient to produce another incentive before the
arousal is "used up" or behavior motivated by the incentive will cease. To
get the simulation going for Killeen's model, I had to begin with a
sufficient rate of behavior already in place to produce the first incentive;
that is, (returning to my engine analogy) I had to wrap the cord around the
flywheel and give 'er a pull. Of course, in real operant studies, there is
always some baseline rate of the instrumental behavior to begin with, or the
animal would never discover the contingency. However, an animal that had
learned to earn food by lever-pressing would probably just approach the
lever and begin pressing, unless in extinction the system had reorganized away.
Killeen's model seems to be fairly successful in accounting for a number of
effects described in behavioral economics, although it may not do so well
once the correct derivations are produced from it. I am encouraged that a
real, honest-to-goodness control-system analysis of the situations to which
Killeen applies his model may yield some important breakthroughs and intend
to pursue this application in the near future.
One more comment. I was encouraged to take a second look at Killeen's
analysis by Chuck Tucker, who thought after reading Killeen's paper that it
contained some elements compatible with a PCT approach. Chuck, you were right!
Reference: Killeen, P. (1995). Economics, ecologics, and mechanics: The
dynamics of responding under conditions of varying motivation.
_Journal of the Experimental Analysis of Behavior_, _64_,
405-431.
Regards,
Bruce