Catch-22

CSGnet Archive CSG1995
1

[From Bruce Abbott (950608.1300 EST)]

Bill Powers (950607.2010 MDT) --
    Bruce Abbott (950607.1155 EST)

    In the absence of responding, each shock terminated automatically
    at 15 seconds. For a rat able to keep the average shock length at
    150 milliseconds (0.15 seconds, control reduces the disturbance
    (shock) to 1% of its uncontrolled value. By your definition this
    is excellent control, yes?

Depends. A formal observational definition of a reference level for a
controlled variable is that level of the variable at which the system's
action just drops to zero. To find the reference level for the shocks,
you'd have to do something to reduce them until the rats wouldn't bother
to act to avoid them. Then you could judge the loop gain by seeing how
much you would have to raise the shocks again to get the maximum amount
(speed, whatever) of action against them. That would define the range of
control, and you could then guess how important a given level of shock
is to the rat by measuring the ratio of error to effort. I suppose this
would entail adjusting the shock current, since the rats have a minimum
reaction time and you don't know if they've reached it.

Studies something of the nature you describe have been done, although I
would like to see them done better in the way you describe. Shock
aversiveness is proportional to something like a weighted product of
intensity and duration, with intensity having the greater weight. At the
intensity I used in this study I think its safe to assume that reference for
shock duration is lower than they were able to achieve.

From one viewpoint the rat is controlling shock duration, presumably to some

reference at or close to zero. This control is very effective in that shock
duration is reduced to something on the order of 1% of its uncontrolled
duration. You seem to be suggesting that the residual 1% may be so aversive
that 1% and 100% are basically equivalent from the rat's point of view. If
that is true, why are they controlling at all?

   In the avoidance experiment the rat performed on a Sidman shock-
    avoidance schedule on one session and the actual temporal pattern
    of shock delivery was recorded (rats on this schedule occasionally
    make mistakes and receive shocks); this pattern was "played back"
    on the next session, in which the rat had no control over shock
    delivery. As with escapable versus inescapable shock schedules,
    the rats failed to resist the disturbance when the apparatus
    switched them from avoidable to unavoidable shock schedules or vice
    versa. The key here is that during training the rats had learned
    that the shock frequency was the same whether they controlled shock
    delivery let the apparatus determine when shocks would be
    delivered.

    Will that do?

Depends. Did the rats demonstrate that they knew how to switch from one
schedule to the other?

Yes (see today's post from me to Bill Leach relative to the avoidance
study). In the escapable- versus inescapable-shock study, I parametrically
varied the duration of inescapable shock across sessions (the duration was
fixed within-session). When the inescapable shocks were about 100
milliseconds or more longer than the _average_ duration of the escapable
shocks, the rats preferred the escapable-shock condition. When the reverse
was true they preferred the inescapable-shock condition. When the shock
durations in the two conditions matched within 100 milliseconds, preference,
if any, was below the sensitivity of the experiment to detect. The rats
knew what they preferred, and they knew how to get it.

Also, I presume that whenever the "controlled" condition appeared, the
rats reduced the experienced shock rate by doing whatever they had to
do. Perhaps their failure to differentiate indicates that whatever the
shock rate was in the controlled condition, it was below the level they
considered an unacceptable amount of error. In the experiments I
analyzed, the rats were holding the shock rate as low as a few shocks
per hour. We could explain the failure to switch conditions as
indicating that the shock rate achieved in the controlled condition was
experienced as zero error. The same shock rate, recorded, would also be
experienced as zero error, or small enough not to bother with.

This, of course, opposite to your earlier suggestion that the rats did not
prefer control because it left and UNACCEPTABLE level of error. So either
way, you propose that the experiment fails. Catch-22.

If the
rats were hovering over the bar in the "control" condition, they must
still have been experiencing a pretty large error.

    What? If your participants keep their hands on the mouse during
    the compensatory tracking task, "they must still have been
    experiencing a pretty large error"? Hmmm.

Depends. If they're leaning toward the screen and moving the mouse as
fast as they can, I assume they still think the errors they're seeing
are pretty large.

Depends. They could be nearsighted and doing an excellent job of correcting
some rapidly varying disturbances. [Hey, I can play this game, too! (;->]

You asserted that the rats MUST be experiencing (strong residual) error
because they were staying on the lever. My example shows that this
assertion is wrong: they COULD be doing an effective job of controlling.
That they might not be is irrelevant in this context.

    Even in the escapable versus inescapable shock experiment the rats
    had no difficulty distinguishing when they did and did not have
    direct control over shock duration. It's pretty obvious: in the
    escapable shock condition the escape lever is extended and, unless
    you press it, shock just keeps goin' and goin' and goin' (up to 15
    seconds). In the inescapable shock condition the lever is
    retracted (so there is nothing to press) and the shock stops after
    a brief, fixed duration, independent of behavior. z

Depends. It's pretty obvious to you, but it is obvious to a rat? Suppose
that to a rat, a 100 millisecond shock (of the intensity used) creates
95% of the maximum possible sensation of being shocked, and any shock
longer than 300 milliseconds raises the perception to the maximum value
the perceptual signal can have -- 100% of the maximum sensation. That's
a 5% different in perception. So whether the shock lasts 300 msec or 15
seconds makes no difference in the perceptual signal (or the error, or
the output): it's saturated.

Research is not conducted in a vacuum. There's plenty of independent
evidence that rats can discriminate such differences, including the internal
evidence from the experiment itself regarding discrimination of shock
duration differences. Psychophysical studies show that saturation is not a
problem: perception of shock duration follows Weber's Law.

    That depends on whether they were in the escapable or inescapable
    shock condition at the time. In the former condition, the shock
    duration was clearly under the rat's control.

Depends. The rats were clearly _trying_ to control the shock when they
could.

I think we're splitting hairs here. They clearly were doing more than
trying. Note what happens to shock intensity when a response occurs: it
instantly falls from a rather aversive level to zero.

The only way to verify that the system is
really controlling is to vary the disturbance and see if the effort
varies in opposition to it. If you find that an increase in shock
intensity results in a substantially shorter reaction time by the rats,
then you know that they were controlling, rather than just trying to
control.

If you are correct, then there is no way to test for control of a logical
variable when the output is a step function.

I don't want you to get the impression that I miss your point. Control over
shock termination is not ideal because it cannot prevent the shock from
occurring in the first place (especially when shock is programmed as a step
disturbance as it typically is in such experiments). I had the same concern
you have raised. When rats turned out not to prefer the escapable shock
condition, I worried that it was because the form of control was not "good
enough." The avoidance experiment, which permitted the rats to prevent
shock from occuring at all, was conceived to address this issue.

Regards,

Bruce

2

Tom Bourbon [941123.1710]

[From Bruce Abbott (941123.0950 EST)]

Bruce, I don't have time right now for the detailed reply your post
deserves. When I asked what kind of thing a probability is that it can be
controlled in a control model, I did not intend to have you playing Catch
22. You defend your inclusion of controlled probabilities on the grounds
that we know little about the physiology of organisms that learn for us to
construct more realistic models of them. (Incidentally, I probably have
less optimism than you about the implications work with aplysia.) One of
the things that bothers me about such an approach is that it forces into the
behavior exactly the appearances it should have -- if the probabilities of
various actions change during "learning," then add precisely those changes
in probability to the output of the model; if the RMS error of tracking
changes when, in mid run, a person shifts from one kind of tracking to
another, then add precisely those RMS errors to the output of the model;
and so on.

More should be said, which probably means I should not have said the little
I did here. Time to go. If the thread is still alive next Monday, I'll try
to do a better job.

Later,

Tom

3

[From Bruce Abbott (941123.0950 EST)]

Tom Bourbon [941122.1743]

In your model, "probability" seems to be controllable entity. It looks to
me as though you include in the model a feature that guarantees the outward
appearances of behavior will be right, but the added feature implies some
(to my thinking) unlikely properties of the system (E. coli).

In your model, what is this thing called "probability?"

One downside of this mode of communication is that each post--and in some
cases each paragraph within a post--tends to be treated in isolation. Your
question was addressed in my earlier post (941121.1830 EST) as follows:

     The mechanisms specified by traditional reinforcement theory are
     admittedly vague (association by contiguity somehow increases response
     probability) and thus difficult to relate to specific physiological
     mechanisms. However, my representation of learning in ECOLI4a is
     precisely as specified by the law of effect: alteration of response
     probabilities as a function of experience. The physical mechanism is
     unknown, so its input-output relationships are modeled instead (i.e.,
     functional relationship).

Lack of knowledge about the detailed structure of a system need not halt
progress in understanding the behavior of a more complex system built on the
lower one if the relevant functional properties of the lower-level system are
known. For example, detailed knowledge of the properties of atoms was not
required in order to model the behavior of a gas in an enclosed vessel.

So how are we to model the changes in response frequency that come about as a
product of experience with the particular environment in which the organism
finds itself? We do not know what physical changes take place in the nervous
system, although recent and current research (e.g., on aplysia) is beginning
to provide some possible answers. We can observe, for example, that under
some conditions "experience" alters the quantity of neurotransmitter released
across the synaptic junction, leading to a increased likelihood that a given
stimulation of the sensory fiber will fire the motor neuron to which the
sensory fiber is linked. In the absence of such detailed knowledge, I can
still model the changes that take place in response probability AS changes in
response probability. The probability mechanisms in the model thus replace
the unknown physical mechanism with functional relationships that affect
behavior, to a first approximation, the way the physical mechanism does.

appearances of behavior will be right, but the added feature implies some
(to my thinking) unlikely properties of the system (E. coli).

When I started down this path, I modeled e. coli as I thought a reinforcement
theorist might, given that no learning is involved. In this case
reinforcement could only act as an "attractant"; the correct responses to the
attractant would have to be built into the organism. However, when I
presented this model, I was informed that it was not a "reinforcement" model
because e. coli did not learn. So I went looking for a reinforcement model of
e. coli that would demonstrate learning. After several rather unsuccessful
attempts to develop a model that DOES learn, I succeeded with ECOLI4a, only to
be informed by you (above) that this model is unsatisfactory to you because it
attributes some unlikely properties to e. coli! You have me playing Catch-22.
It is, of course, not a model of e. coli, but of some hypothetical organism
with the behavioral abilities of e. coli (swim and tumble) but also capable of
trial-and-error learning. It MAY be a reasonable model of the human
participants in the Powers/Marken e. coli study. More to the point, it
demonstrates a fast and efficient method whereby organisms with sufficient
mental capability (certainly NOT e. coli) learn to control their perceptions.

Regards,

Bruce