[From Bill Powers (941114.2200 MST)]
Bruce Abbott (941114.1630 EST) --
Got your new code for ecoli4a, compiled it, and ran it. An interesting
feature that you may have noticed: ps+ (pardon my shorthand) approaches
zero (actually 0.005 but let's call it zero) and ps- approaches 1. This
happens relatively quickly.
Suppose we start with these probabilities at their limits. Then we can
understand the following code segment very easily:
if dNut > 0 then { S+ present; tumble probability determined by S+ }
begin
if (Random < pTumbleGivenSplus) then DoTumble
else JustTumbled := false;
end
else { S- present; tumble probability determined by S- }
begin
if (Random < pTumbleGivenSminus) then DoTumble
else JustTumbled := false;
end;
NutCon := NewNut;
Note that in the limit, pTumbleGivenSplus goes essentially to zero, and
pTumbleGivenSminus goes to 1. This means that in the first clause of the
overall "if" statement,
if dNut > 0 then { S+ present; tumble probability determined by S+ }
begin
if (Random < pTumbleGivenSplus) then DoTumble
else JustTumbled := false;
end
... there will never be a tumble (that is, "random" will never be less
than 0). Thus if dNut > 0 there will never be a tumble -- until the path
passes a right angle to the target and dNut becomes negative.
In the second clause
else { S- present; tumble probability determined by S- }
begin
if (Random < pTumbleGivenSminus) then DoTumble
else JustTumbled := false;
end;
.... pTumbleGivenSminus is 1 in the limit, so when dNut <= 0, there will
always be a tumble immediately.
Therefore, when the probabilities reach their limits, the above code
segment is closely equivalent to
if dNut <= 0 then DoTumble.
Now we can understand why the final approach to the target is so rapid
and the final position stays so close to the target: the model has
approached the condition in which there is a tumble for any movement
down the gradient and none for any movement up the gradient, as in the
simplest PCT model.
···
------------------------------------
The question is now why the two probabilities tend so rapidly and
systematically toward 0 and 1. The implication is that NutSave, the
previous value of dNut, predicts the next value of dNut. But we know
that this is not true. Random means random: whatever the current value
of dNut, the next value can be anything from the maximum positive to the
maximum negative, and the most probable value is zero.
The logic here is extremely complex, but there is a simple way to see
whether the previous value of dNut is really acting as a reinforcer.
Change the sign of dNut that is saved as NutSave. That is, in
procedure DoTumble;
begin
Tumble(Angle);
JustTumbled := true;
NutSave := dNut; { NutSave is nutrient rate of change
immediately }
end; { after a tumble
}
change the last statement to NutSave := -dNut.
We see the probabilities change in the same directions as before,
although now they do not come as close the the limits as before. The
model still progresses toward the target.
Going even farther, we can write
NutSave := dNut * 2.0* (random - 0.5);
... with the same result, only now the probabilities do eventually reach
the limits of 0.005 and 1.0.
The quickest results of all come from simply randomizing NutSave:
NutSave := random - 0.5;
So what is making the probabilities change is not any systematic effect
of NutSave, but something else about the nonlinear and circular geometry
of this situation, combined with the complex logic. I really don't have
the faintest idea why the net effect is swimming up the gradient. The
situation is too complex for me to see how to apply control theory. But
it is clear that the reason is NOT an effect of the previous value of
dNut, or of the change in value across a tumble.
--------------------------------------------
I don't like complex models for this very reason. Human logic (at least
my human logic) is simply not up to the task of understanding what such
models do. Your explanation of how the model works seemed perfectly
plausible to me at first, until I started wondering how the model
achieved the effect. Then the more I tried to see what was happening,
the less I understood. Finally I gave up and just challenged the concept
that the previous value of dNut was having a systematic effect on the
probabilities. As soon as I did that by introducing arbitrary changes in
NutSave, and got the same result as before, I could see that the
explanation couldn't be right -- but I'm damned if I can figure out what
the right explanation is.
----------------------------------------------
Which brings us back to Rick's original reason for publishing this E.
coli effect: when the tumbles are random, all links between previous
behaviors and current behavior are cut off. Yet the model (and E. coli)
approaches the target very efficiently. When previous consequences
cannot have any systematic effect on future behaviors, reinforcement
theory can't work, but perceptual control theory does work (where we can
see how to apply it). So goal-seeking behavior is shown, in this
instance, NOT to be explainable in terms of reinforcement effects on
behavior.
Showing that reinforcement involves a misinterpretation in other
contexts requires other methods, some of which I have incorporated in
those phenomena in the list of requested explanations I posted a couple
of days ago. I think it's important that we go through those effects and
see what reinforcement theory can do with them. I think we are going to
find that reinforcement theory doesn't explain these phenomena either,
or predict them correctly.
I assume, of course, that there IS such a thing as "reinforcement
theory", a basic set of statements about reinforcement from which we can
deduce what reinforcement theory would predict in various situations.
------------------------------------------------------------------------
Best,
Bill P.