Thorndyke's cats etc.

[From Bill Powers (951110.0900 MST)]

Martin Taylor 951109 17:00 --

It looks as though we agree on the fact that the puzzle-box cats will
cease to reorganize when they find the first variable to control that
will reliably result in escape. This leads to stereotyped behavior. We
also agree that the variables they end up controlling may not have any
connection _known to the cat_ with whatever actually causes the box to
open.

If the box is actually to open, however, something about the controlled
variables being in a certain state or the actions involved in
controlling them must have the required physical effect on the release
mechanism. Since there are many variables a cat could control that would
entail applying pressure to the release mechanism, different cats will
settle on controlling different variables. What all these control
processes have in common is that they entail, somewhere in the process,
applying pressure to the release rod.

The very stereotypy of the cat's behavior points us toward the variables
that the cat is controlling. If we rearranged the environment to
interfere somewhat with the stereotyped behavior (but not with its
effect on opening the box), the cat should make countering adjustments
so the same stereotyped behavior is still produced. For example, if the
cat lies on its back next to the release rod, we could require that it
take a circuitous route to that position and see if it still ends up in
the same place and orientation.

A slight diversion:

Here is how Thorndike described the process in _1898_(Watson, R.; _Basic
writings in the history of psychology_):

     When put in the box the cat would show evident signs of discomfort
     and of an impulse to escape from confinement. It tries to squeeze
     through any opening; it claws and bites at the bars or wire; it
     thrusts its paws out through any opening and claws at anything it
     reaches; it continues its efforts when it strikes anything loose
     and shaky; it may claw at things within the box. It does not pay
     very much attention to the food outside, but seems [s]imply to
     strive instinctively to escape from confinement. The vigor with
     which it struggles is extraordinary. For eight or ten minutes it
     will claw and bite and squeeze incessantly. (p. 255)

This reminds me of our final trip from Illinois to Colorado, when we
brought our cat along. The cat had always resisted being put in the car,
so thinking to insulate it from the moving scenery and the car's
interior, I prepared a heavy cardboard box with little windows, a
familiar blanket, food, and water, popped the cat in at the last second,
and closed the lid. Mary and I drove off. Immediately there was bumping
and meowing and rocking in the box on the back seat, and in a short time
the top of the box heaved open and the cat got out. After a stop to haul
the cat out from under a car seat, put it in the box, and pile another
box on top of the lid, the trip continued. For a couple of hours there
were more bumps and meows and scrabbling, and claws and teeth at the
edges of the little windows, and by the time we had been on the road for
three hours (Mary says less), the cat had shredded a great hole in the
side of the box, escaped from it, and disappeared under the driver's
seat (an even more confined space) where it remained for the next day
and a half. Note that the food and water were _inside_ the box, and the
cat, as per plan, had not yet eaten that day. I had to get them out and
put them on the floor behind the driver's seat to keep the cat from
starving, at least in my imagination. My ultimate reasoned conclusion
was that I didn't know what the hell this cat was controlling for.

Actually, this behavior fits Thorndike's description better than many
theoretically-driven paraphrases of his findings seem to do. Being in
the box was itself an unacceptable situation for the cat. Thorndike says
his cat didn't pay much attention to the food outside the box; mine paid
_no_ attention to the food and water _inside_ the box. If the sight and
smell of the food had been the primary motivator, my cat would not have
tried to escape both from the box and the food. It's clear that
Thorndike had the idea that food outside the box would give the cat a
reason to try to escape from the box (why else put it there?), but even
he had to admit that the cat wasn't particularly yearning toward the
food, but was simply attempting to get OUT OF HERE.

How could Thorndike have gone from these quite honest observations to
his "law of effect?" I think the answer is that Thorndike's eye was on a
different goal. In the following he reveals that he has a number of
other axes to grind:

     The cat does not look over the situation, much less _think_ it
     over, and then decide what to do. It bursts out at once into the
     activities which instinct and experience have settled on as
     suitable reaction to the situation "confinement when hungry with
     food outside." The one impulse, out of many accidental ones, which
     leads to pleasure becomes strengthened and stamped in thereby, and
     more and more firmly associated with the sense-impression of that
     box's interior. ... Futile impulses are gradually stamped out. The
     gradual slope of the timecurve, then, shows the absence of
     reasoning. They represent the wearing smooth of a path in the
     brain, not the decision of a rational consciousness. (p. 256).

There's somebody standing just offstage with whom Thorndike is arguing.
Thorndike is saying "See? The cat does NOT look over the situation. The
cat does NOT think. The escape is NOT purposeful, but an accidental
result of being driven by impulses. The cat is NOT trying to escape from
a terrifying situation, but is being driven to attain pleasure. The cat
does NOT make decisions; the cat does NOT have any rational
consciousness. So you're full of garbage!" When we read Thorndike, we
have stepped into the middle of a fight. Of course his arguments don't
refute any of the contrary notions he seems to be trying to dispel; he
says himself that he knows of no mechanisms to achieve these effects, so
how he knew that these mechanisms don't involve thought etc. is beyond
me. But he seemed to think he had made his case.

The story Thorndike ended up telling was this. There is food outside the
box. The cat goes through a series of instinctive or previously-acquired
behaviors until it accidentally opens the box, whereupon it is able to
go to the food and eat it. "The writer (1898, 1914 and 1931) has
maintained that the after-effects of a modifiable connection work back
upon it, and that, in particular, a satisfying state of affairs
accompanying or directly following a connection strengthens it." (p.
257). So eating the food acts upon the connections giving rise to the
behaviors that open the box to strengthen those that led to reaching the
food and "stamp out" those that didn't. Immediately after that passage,
we discover at least some of those with whom he was arguing:

     The great majority of psychologists have maintained, on the
     contrary, that the strengthening of any connection is due to forces
     operating within the connection itself or prior to it. (p. 257)

A pity that Thorndike didn't have the concept of a reference signal -- a
"force" that definitely operates prior to the establishment of any
connection. I think we have to keep in mind that Thorndike would have
rejected as mystical any suggestion that the cat _wanted to be out of
the box_. That would have struck Thorndike as a mentalistic explanation,
since he, like most scientists of his time, saw a division between
"mental" and "physical" phenomena. Thorndike would also have scoffed at
the idea that an electrical circuit could prove theorems.

···

========================
Anyhow. Are we coming any close to solving your problem, Martin?
-----------------------------------------------------------------------
Bruce Abbott (951109.1820 EST) --

     Guthrie, who offered an alternative to Thorndike's "Law of Effect"
     explanation, held that what "selected" the behavior was not
     reinforcement but the fact that the behavior was immediately
     followed by a change in the situation, i.e., the cat left the box.

I think Guthrie was closer than Thorndike. Did anybody ever repeat the
experiment with the food inside the box?

What bothers me the most about either Thorndike's or Guthrie's
proposition is that we have an abstract fact having an effect on a
connection (presumably in the nervous system) which is not even active
any more by the time the result occurs. If you tried to build something
that would work like this you'd end up with a complex mechanism
containing memory and all sorts of other functions, and the real
explanation would obviously lie in those mechanisms, not in the
environment.
-----------------------------------------------------------------------
Chris Cherpas (951109.1502 PT) --

     If I get around to it, or if you do a simulation of Conc VIs, I
     recommend linear VIs (a la Vaughan).

I wish you guys would remember that you're talking to ignoramuses here,
who can't necessarily stroll over the the library and look up the things
you allude to (who the heck is Vaughan, and what did he do, and what is
a linear VI, and what other kinds of VIs are there?)

     Not that it makes a huge difference, but if somebody gets that idea
     that the controlled variable is session/overall rate of
     reinforcement, then they might think that matching is explained by
     this.

That is surely _one_ controlled variable. One way to model choice
behavior is to use one control system that raises and lowers the
reference levels equally for the control systems associated with each
choice, to control the total food intake, and another which varies the
allocation to maximize the overall ratio of reward rate to behavior
rate. You have to account somehow for the fact that there is any
behavior at all, on either alternative key. You can have the same
relative allocation at any level of total intake.

My assumption would be that if matching does occur to some degree, it is
a side-effect of controlling some more basic variable such as the ratio
of benefit to cost. I can't see organisms controlling matching for its
own sake. You have to ask what's in it for the organism, and use the
answer as a clue to the real controlled variable.
------------------------------
     Finally, Bruce, do you have any doubt that random responding on
     Concurrent VIs would produce anything other than severe
     undermatching (flat or even slight "anti-matching" -- negative
     slope?).

Well, Bruce may not have any doubts and I might even not have any
doubts, but in PCT when you make statements like that you SHOW that they
are true. You can't get away without doing the actual simulation. We are
a hard-nosed bunch when it comes to making predictions. No shortcuts
like "Can't we just agree that this is what would happen?" When you
actually do simulations, you always discover something you hadn't
expected.
-----------------------------------------------------------------------
General comment: Yesterday the system managers at Fort Lewis posted a
notice saying that the internet was experiencing routing problems,
particularly in the East, and that there was no projected time by when
the problem would be solved.
-----------------------------------------------------------------------
Best to all,

Bill P.