control of behavior by consequences; preliminary test

[From Bill Powers (941030.1845 MST)]

Bruce Abbott (941029.1700 EST) --

Got your program to run with no trouble; grUtil is now a unit in my


Obviously there can be no effect on the current behavior by
its consequence, the reinforcement that has not occurred yet and may
or may not occur. Only after the behavior has taken place (swim for
some length of time and then tumble) and the consequence occurs can
the consequence be known

I disagree with you here. Not swim AND THEN tumble, but merely swim.
The behavior on which reinforcement would act in this context is
forward motion along a given trajectory.

Let's examine your code, which I reproduce here with its parts labelled
according to PCT:

procedure StepEColi;
  NewNut: real;
  EcoliX := EcoliX + Speed * cos(Angle); |
  EcoliY := EcoliY + Speed * sin(Angle); |
  X := Round(EcoliX); |
  Y := Round(EcoliY); ENVIRONMENT
  PutPixel(X, Y, white); |
  NewNut := NutConcen(EcoliX, EcoliY); |


  dNut := DecayRate * dNut + (NewNut - NutCon);
  if dNut <= 0 then COMPARATOR
  NutCon := NewNut; {program process}

OK, what is the "consequence" and what is the "behavior" here? I think
we can probably agree that dNut is the consequence; it's a perceptual
representation of the time rate of change of nutrient concentration just
outside the thingie. What is the behavior? I have a little problem with
this because it seems to be the execution of a tumble the instant that
dNut falls below its reference level of zero. It's hard to see how the
behavior could be affected by the consequence -- how could the behavior
change if it is always the execution of a tumble? About the only
dimension in which this behavior could change would be in rate, or in
number of tumbles needed to get within some distance of the goal.

So I added a termination condition, ..." or (Dist < 3)", and made Dist a
global variable.

Then I supplied a routine for showing real numbers on the screen:

procedure ShowReal(x,y: integer; v: real);
var s: string;
bar (x,y,x+textwidth(s),y+textheight(s));

And I changed the procedure for stepping:

                    ======================= added
  if (dNut <= 0) or (random(1000) < 100) then
    NumTumbles := NumTumbles + 1.0;
   showreal(maxx - 100,20,NumTumbles);
   showreal(maxx - 100,40,dNut);

Also, you will notice, I added a disturbance to the output, by creating
random tumbles independently of the value of dNut. If the consequence
were actually controlling the tumbles, disturbing the tumbles should
result in a change of dNut that would oppose the change in tumbling, and
restore it to (or toward) the undisturbed state.

As far as I can see, that doesn't happen. When you put in the
disturbance, the number of tumbles needed to get near the goal goes way
up (a factor of 10 with the numbers I used), and dNut increases much
more slowly.

On the other hand, when you add a disturbance to dNut (instead of the
above disturbance -- remove it):

                                                ====== disturbance
  dNut := DecayRate * dNut + (NewNut - NutCon) - 0.01;

the value of dNut just continues to rise as the target is approached,
much as without the disturbance.

This is all very rough and ready, and I'll try to do a more careful job.
But at the moment it looks as if dNut is controlled, and the rate of
tumbling is not controlled.

Bill P.