Angle criterion

[From Bill Powers (960928.0630 MDT)]

Martin Taylor 960927 13:40 --

More to the point, you made a statement that when you, Bill P., tracked
with the pendulum task yesterday, you got a model fit of 98% correlation
or thereabouts, which is much better than I ever got. You _did_ these
tracking runs, and you have the data and the analysis routines that would
let you compute the angle criterion for these very runs. All you have to
do is to report it. To do that should add perhaps half an hour to the
hundreds you have put in on the study.

I don't want to do it, Martin. I am already satisfied that the model fits
the performance of a practiced subject better than an ideal control model
would fit it. I don't know of any other control models that would work
better, so questions that deal with "all possible control models" are empty
until some alternative model is proposed and tried. While there are many
ways to implement equivalent control models, all models having the same
transfer function are mathematically redundant and can't be distinguished
experimentally because they would all perform exactly the same way. You're
talking about models with _different_ transfer functions, and as I say I
don't know of any -- if I did, I would be using the one that performs best.
Your angle criterion, so far, is being applied to a sample of one model.

Why not buy a computer that can run the experimental programs? I am on my
fifth computer, and I have bought all of them myself, two of them since I
retired, because I wanted to run these programs. I am no better a programmer
than you are, although I am MUCH better at estimating the time it will take
for someone else to complete a project. If you really think your question is
important, you will take the time and spend the money to find the answer.
That's what other people do.

Your argument that since I have already invested so much time in this
dubious project, I ought to be willing to invest more, is the same argument
that leads boards of directors to drive their companies into bankruptcy. I'm
sure you've read reports of studies on this well-known phenomenon.

Best,

Bill P.

[Martin Taylor 960930 13:55]

You _did_ these
tracking runs, and you have the data and the analysis routines that would
let you compute the angle criterion for these very runs. All you have to
do is to report it. To do that should add perhaps half an hour to the
hundreds you have put in on the study.

I don't want to do it, Martin.

I don't understand why not, when the tools and data are immediately at hand.

A week or two ago, you said that the angle criterion (or rather, a version
that looked at the wrong angle of the triangle) would probably be a better
criterion than the correlation criterion. Separately, you said that any
results from the sleep study would be invalid because the subjects were
not as well trained as you yourself, and you quoted your own results on
the pendulum test to back this up. One might expect you to _want_ to use
the angle criterion on your own data.

I have found the results of the sleep study to show a dissociation between
the values for the correlation criterion and for the angle criterion across
the different tasks--and I have now checked this out with the smallest
amount of averaging I can achieve easily (using only the four G and U runs
for each task for each subject separately during the day after the recovery
sleep, when the subjects were again fresh and had learned all they could
from their 400+ prior tracking runs). There is no overlap--none--between
the area covered by the 34 points in the pursuit tracking and in any of
the other tasks. Within the Placebo and modafinil subject groups, there
are overlaps at the edges of the areas covered by the pendulum, the
compensatory tracking and (together) the number-at-50 and the disk-on-circle,
but for the amphetamine group these three regions are again non-overlapping.

The number-at-50 and the disk-on-circle distributions have much the same
centroid, but there is one very surprising fact: across all the 34 subjects,
the scatter is smallest for the number-at-50 task, with a high correlation
(centred about 0.97) and a moderately low angle value (centred about 0.8).

Now, these results may well be because the subjects were not controlling
the same perception you would have been controlling, especially in the
pendulum task. Perhaps they were trying to control two perceptions at once,
with some conflict--which would demand a different model for the subjects,
if not for the individual control loops within a subject. Who knows? But
if you get the same low angle value (again around 0.8, but differing
markedly across subjects from 0.7 to 0.95) despite a high correlation,
then there's something more going on.

Remember that we are dealing in all cases with the same model, a pure
integrator output stage, a fixed loop delay, and a derivative of the
perceptual signal added to the reference value. Only the parameters are
changed so that the RMS deviation between the model's behaviour and the
human's is minimized. The correlation is between the human's data and the
model's, and the angle is from the ideal to the human to the model, scaled
to make a right-angle = 1.00. If the angle is less than 0.5, the ideal
(i.e. the _fact_ of control) is having more influence on the model than
is the human tracking data. I have never found such a case in the small
part of the data I have scanned, but 15 of the 34 subjects have angles
below 0.7 in the compensatory tracking task.

You say that the results are unreliable because of the nature of the study.
The more I look at the results, the less I agree with you, but that's
neither here nor there. What would be nice is if you could see whether the
same or similar results hold for a subject in whose performance you trust:
yourself. Since you have both the data and the analysis program, I am
asking you to invest very little except your faith.

I am already satisfied that the model fits
the performance of a practiced subject better than an ideal control model
would fit it.

Yes, we've known that for as long as I've been on CSGnet, and you've known
it for decades. The experiment is not intended to test the counter-assertion.

I don't know of any other control models that would work better,

Well, in order to get the good fits, we _did_ have to construct a control
model that worked better than the simple integrator-with-delay loop, so
in fact you do know of a control model that works better. A full two-level
model might work better yet--or maybe not. There are lots of other
possibilities.

While there are many
ways to implement equivalent control models, all models having the same
transfer function are mathematically redundant and can't be distinguished
experimentally because they would all perform exactly the same way.

And they wouldn't be very interesting, would they?

You're
talking about models with _different_ transfer functions, and as I say I
don't know of any -- if I did, I would be using the one that performs best.
Your angle criterion, so far, is being applied to a sample of one model.

That's right. And with the pursuit task, according to _both_ criteria, it's
hard to bet that we could find a model that would work better. That's the
pursuit tracking task on which so much of the weight of evidence rests
with respect to PCT. On all the other tasks, the angle criterion suggests
that a better model waits to be found, at least for the sleep study subjects.

Is this also the case for a highly practiced and motivated subject?

Why not buy a computer that can run the experimental programs?

Lots of reasons, divorce being not the least. But for me to make such an
investment when you can so easily provide an answer would be stupid at best.

If you really think your question is
important, you will take the time and spend the money to find the answer.

It is important within one context. Not within the context of how much Ina
would like to spend landscaping the garden and I would like not to spend
fixing the rook leak :frowning: And buying software for the computer I _did_ buy
for my own purposes--a Mac clone.

But I would have thought that such a question would have been fairly critical
to you, though. If it turns out that for you as S the angle criterion is near
unity for all tasks, it would be pretty strong support for your claim that
the sleep-study subjects were operating during a phase in which they were
still learning what to do. If not, and your results agree with theirs for
at least some of the tasks, then we will have found a way of approaching
the long-sought tool for teasing apart some of the aspects of control
structure about which so much has been said over the years. Maybe this
is true anyway. And again, maybe not.

Your argument that since I have already invested so much time in this
dubious project, I ought to be willing to invest more, is the same argument
that leads boards of directors to drive their companies into bankruptcy.

That's really an unfair, irrelevant, untrue, and ad hominem comment.

My argument is that this "dubious project" has raised a question that
you can answer by using, with very little effort, the work that you have
already invested. That's diametrically opposed to what you say. Although
I dispute your "dubious" statement, nevertheless for the purpose of
argument I can go along with it, asking you now to do something that
you _cannot_ call "dubious," since you will be using the one and only
subject (or perhaps one of the three) whose results you trust.

As I understand it, you are always looking for ways to test both PCT and
the specific models applied to particular tasks. I am puzzled by your
reluctance to do so.

Martin

[From Bill Powers (960930.1330 MDT)]

I don't want to do it, Martin.

I don't understand why not, when the tools and data are immediately at hand.

...

As I understand it, you are always looking for ways to test both PCT and
the specific models applied to particular tasks. I am puzzled by your
reluctance to do so.

It isn't necessary that you understand my reluctance in order for it to
exist. But perhaps if you just push harder, you can get me to relax my
resistance to doing the task you want to set for me.

Best,

Bill P.

[Martin Taylor 960930 16:10]

Bill Powers (960930.1330 MDT)

It isn't necessary that you understand my reluctance in order for it to
exist.

True, but if I understand what you are controlling for, it helps me to
find a disturbing influence that might help me to get my perceptions nearer
their reference values through your actions.

But perhaps if you just push harder, you can get me to relax my
resistance to doing the task you want to set for me.

Harder, or differently? Let's try differently.

I believe I sent you each of my series of analysis c-code, all of which
were based on your own code, with amendments suggested by both of us.
All the later codes (and perhaps all the earlier ones as well) produce
outputs for the mean square deviations between: human and target (ht), human
and model (hm), model and target (mt).

All I want you to do is to perform a tracking run (even one good run might
do, though two or three would be better) for each of the five tasks, and
then to do the analysis, getting out the correlation fit and the value:
xx = (4/3.14159)*atan2($12,$11);
where $12 and $11 are the twelfth and eleventh outputs of the newer
analysis programs--result.pe1 and result.te1 in all the programs, I think.
These are the RMS prediction error (which is minimized by the fitting
algorithm) and the RMS tracking error (the actual data).

The fitting part of the model I am using follows, in case you need
to identify which one corresponds to my data--though it would be interesting
indeed to try the same thing with the various models we have used.

I hope that this will suggest to you that only a small investment of
your time is required, and that the results might be useful--if only to
shut me up, but perhaps for PCT as well.

Martin

----------------C code follows, to help identify which analysis------------
---------------program I have been using in the reported results-----------

void fitparams()
{ double errsqud, errsquk, errsquz, errsqdd, errsqdk, errsqdz, lasterrsq;
  double testupz, curincz, testupk, curinck, testupd, curincd;
  double cincd, cinck, cincz;
  double testdnd, testdnk, testdnz;
  double ediffud, ediffuk, ediffuz, ediffdd, ediffdk, ediffdz, ediffsum;
  double newd, newk, newz, newerrsq, minerrsq;
  double mewd, mewk, mewz, mewerrsq;
  double stepz, minstepz, stepk, minstepk, stepd, minstepd;
  int i, j, jold, imax, imin;
  short dbl, halve;
  double emin, emax;
  double pwrs;

  /* initialize */

  curincz = startincz;
  curinck = startinck;
  curincd = startincd;
  curz = startz;
  curk = startk;
  curd = startd;
  dbl = halve = FALSE;
  stepd = startincd;
  stepk = startinck;
  stepz = startincz;
  minstepd = 0.01;
  minstepk = 0.01;
  minstepz = startincz/8.0;

  pwrs = runexpt ((float)curd, curk, curz);

  while ((fabs(stepd)>=minstepd)||(fabs(stepk)>=minstepk)||(fabs(stepz)>=minstepz))
{
    lasterrsq = runexpt ((float)curd, curk, curz);

    testupz = curz + curincz;
    testupk = curk * curinck;
    testupd = curd * curincd;

    minerrsq = lasterrsq; j = 0;

    errsqud = runexpt(testupd, curk, curz);
    errsquk = runexpt(curd, testupk, curz);
    errsquz = runexpt(curd, curk, testupz);

    ediffud = errsqud - lasterrsq;
    if (ediffud < 0) {minerrsq = errsqud; j = 1;}

    ediffuk = errsquk - lasterrsq;
    if (errsquk < minerrsq) {minerrsq = errsquk; j = 2;}

    ediffuz = errsquz - lasterrsq;
    if (errsquz < minerrsq) {minerrsq = errsquz; j = 3;}

    ediffsum = fabs(ediffud) + fabs(ediffuk) + fabs(ediffuz);

/* Test at a point suggested by the improvements or worsening along the
/* parameter axes.
*/
    if (ediffud > 0) cincd = 1.0 - (curincd - 1.0)*ediffud/ediffsum;
    else cincd = 1.0 + (curincd - 1.0)*ediffud/ediffsum;
    if (ediffuk > 0) cinck = 1.0 - (curinck - 1.0)*ediffuk/ediffsum;
    else cinck = 1.0 + (curinck - 1.0)*ediffuk/ediffsum;
    if (ediffuz > 0) cincz = 0.0 - curincz * ediffuz/ediffsum;
    else cincz = curincz * ediffuz/ediffsum;

    newd = curd * cincd;
    newk = curk * cinck;
    newz = curz + cincz;

    newerrsq = runexpt(newd, newk, newz);
    if (newerrsq < minerrsq) {
       minerrsq = newerrsq;
       j = 4;
    }

    mewd = curd / cincd; /* Test the opposite direction in case we are */
    mewk = curk / cinck; /* at the bottom of a diagonal trough */
    mewz = curz - cincz;

    mewerrsq = runexpt(mewd, mewk, mewz);
    if (mewerrsq < minerrsq) {
       minerrsq = mewerrsq;
       j = 5;
    }

    if (j == 0) {
     if (halve){
       curincd = 1.5 - (curincd)/2.0;
       curinck = 1.5 - (curinck)/2.0;
       curincz = curincz/2.0;
       dbl = FALSE;
     } else {
       curincd = 2.0 - curincd;
       curinck = 2.0 - curinck;
       curincz = 0.0 - curincz;
       dbl = FALSE;
       halve = TRUE;
     }
    }
    if (j == 1) {
      curd = testupd;
      dbl = FALSE;
      halve = FALSE;
    }
    if (j == 2) {
      curk = testupk;
      dbl = FALSE;
      halve = FALSE;
    }
    if (j == 3) {
      curz = testupz;
      dbl = FALSE;
      halve = FALSE;
    }
    if (j == 4) {
      curd = newd;
      curk = newk;
      curz = newz;
      if (dbl) {
        curincd = 1.0 + (curincd - 1.0)*2;
        curinck = 1.0 + (curinck - 1.0)*2;
        curincz = 2*curincz;
      }
      dbl = TRUE;
      halve = FALSE;
    }
     if (j == 5) {
      curd = mewd;
      curk = mewk;
      curz = mewz;
      dbl = FALSE;
      halve = FALSE;
    }
   jold = j;

    stepd = curincd - 1.0;
    stepk = curinck - 1.0;
    stepz = curincz;

  }

  zband = curz;
  bestd = curd + 0.5; /* round rather than truncate */
  intk = curk;
  lasterrsq = runexpt((float)curd, curk, curz); /* to re-set the model trace for later */
}
#endif

float runexpt(float dl, float k, float z)
{
short i,j,m;
float h,p1,p2,p,e,pold,pz,p3,deltap,u,errsq,errsum, dfract;
float hdelay[31];
short dy;

dy = dl;
dfract = dl-dy; /* get the fractional remainder of the floating delay value */

errsq = 0.0; errsum = 0.0;
for(i=0;i<31;++i) hdelay[i] = 0.0; /* Ring buffer for CEV values */
h = 0.0; m = 0; pold = pz = p = ref; deltap=0.0;

for(i=0; i < datasize;++i) modelh[i] = 0;

for(j= -200;j<datasize;++j)
   {
    if (--m <= 0) m = 30;

    i = abs(j);
    hdelay[m] = h + distsign * (float)dist1[i];

    p1 = hdelay[(dy+m)%30 + 1]; /* Assuming the delay is all perceptual! */
    p3 = hdelay[(dy+m-1)%30 +1]; /* Added by MMT 950929 */
    pz = p3*(1-dfract) + p1*dfract; /*MMT 950929 to centre derivative on same
                                    /* moment as current perception */

    e = ref - p;
    deltap = (pold - pz)/2.0; /* pz is one sample time before p,
                                /* pold one sample time after p */

    h += k*e + z*deltap; /* pold and deltap added 950929 MMT */
    pold = p; p = pz;

    if (h > 1000.0) h = 1000.0;
    if (h < -1000.0) h = -1000.0;
    if(j < datasize && j >= 0)
      {
        modelh[j] = (short)h;
        u = (float)(modelh[j] - realh1[j]);
        if (valid[i]) {
          errsum += u;
          errsq += u*u;
        }
      }
    }
  errsq = errsq - errsum*errsum/validsize;
  return errsq;
}

[From Bill Powers (960930.2000 MDT)]

Martin Taylor 960930 16:10

(WTP)>>It isn't necessary that you understand my reluctance in order for it

to exist.

(MMT)>True, but if I understand what you are controlling for, it helps me to

find a disturbing influence that might help me to get my perceptions nearer
their reference values through your actions.

I don't want to be involved in this project any more, and I don't intend to
be involved. I have no desire to get you to stop working on it, or to
prevent you from taking any data you consider important and analyzing it any
way that makes sense to you. Nor do I want to elaborate further on my
reasons for not becoming engaged in it, because even though they are my own
reasons they could be construed as criticisms, especially if I tried to make
them sound objectively justified.

The problem with applying disturbances to make someone else do what you want
is that if any resistance is aroused, you have already violated reference
conditions in the other person of which you know nothing. Perhaps trying
another kind of disturbance will work, but it's not likely. You can appeal
to pride, to a sense of obligation, to duty, to fear, to ambition, and all
the rest -- but that is only trying to create a conflict such that
acquiescence will be less unpleasant than resistance. The basic fact is that
there is resistance, and if you want to preserve a peaceful relationship the
only true way to get what you want is to look elsewhere or get what you want
for yourself.

The only way to avoid the problems entailed in trying to control other
people is to stop _wanting_ to control them. You can still get what you want
-- but not that way.

Best,

Bill P.

[Martin Taylor 961003]

Bill Powers (960930.2000 MDT)

I don't want to be involved in this project any more, and I don't intend to
be involved.

I find it passing strange that when it is becoming more and more probable
that we might have found a new approach to testing the structural validity
of specific control hierarchies, that is the moment you choose not to
want to play any more.

You did suggest that I push harder, which I understood to imply:

You can appeal
to pride, to a sense of obligation, to duty, to fear, to ambition, and all
the rest -- but that is only trying to create a conflict such that
acquiescence will be less unpleasant than resistance.

That is why I refused then and refuse now to push harder, and why I tried to
propose something different. You referred to an unwillingness to think
further about the sleep study, and I accepted that unwillingness. I did
not interpret it as an unwillingness to test out a possible new tool for
PCT studies.

The only way to avoid the problems entailed in trying to control other
people is to stop _wanting_ to control them. You can still get what you want
-- but not that way.

If one doesn't ask for what one wants, one is unlikely to get it. The problem
arises only when the person asked has a reference for not providing it.
Is there a problem in trying to "control" a waiter by asking for a
selection of items on a menu in a restaurant? I work under the presumption
that the waiter's purpose is to provide the food that will satisfy my
purposes, and that to _refuse_ to try to control him would give rise to some
conflict: "What would M'sieur like today?" "I'm not going to try to control
you, so I won't tell you. Bring me what you want to bring me." "Ah, but
M'sieur may not be happy with what I bring." "I refuse to try to control
you." ... followed by eventual ejection of would-be diner from the
restaurant.

I have, up to now, been working on the presumption that we both have a
reference for understanding more about PCT. I had presumed that when you
were given the opportunity to test out a proposed tool for the purpose,
you would be interested in seeing whether it was a real tool--especially
given two reasons to believe that you would: (1) when I first mentioned
the tool, you said it sounded like a very good approach; (2) you claim
to be one of the very few people who are good practiced trackers, whose
results using the tool you would trust. You even ran a few new tracks to
illustrate the point.

I appreciate that you don't want to do it, and I'm not pushing you to do
it. But I'm surprised and disappointed that you don't think that it suits
your purposes to do so.

As far as I understand it, only Rick Marken and Tom Bourbon are considered
to be good, practiced, trackers whose data would be reliable enough to
validate the tool. I don't know whether they have the programs to run the
tasks, and the programs to analyze the data, as you do, but if they do,
I hereby ask either of them whether they would be willing to do a set
of 10 or 15 runs to see whether the data for good, practiced, subjects
show up the same kinds of things that seem to be persistent in the data
from every subject in the sleep study.

I've discovered one of the sleep-study PCs that still has the software
on it, and isn't being used at the moment (it will be, quite soon).
On it, I have run a couple of tracks myself to see whether everything
still works properly. I haven't yet tried to analyze the data, but it
seems likely to be possible. But I'm not a skilled, practiced tracker
on these tasks, as you are. Whatever my results, they will be subject
to the same criticisms you level at the sleep study subjects.

Martin

PS. I'm away again until probably next Thursday or thereabouts.

[From Bill Powers (961003.1745 MDT)]

I find it passing strange that when it is becoming more and more probable
that we might have found a new approach to testing the structural validity
of specific control hierarchies, that is the moment you choose not to
want to play any more.

I'm not as impressed with your claim about testing structural validity as
you are. I think you're overinterpreting your simple formula, and ignoring
the fact that the behavior being modeled has not yet reached a final form.

You did suggest that I push harder, which I understood to imply:

I was being ironic; as a competent PCTer, you know that when there is
resistance, pushing harder will only lead to more of it.

The only way to avoid the problems entailed in trying to control other
people is to stop _wanting_ to control them. You can still get what you want
-- but not that way.

If one doesn't ask for what one wants, one is unlikely to get it. The
problem arises only when the person asked has a reference for not providing
it.

Or when providing it would cause other goals to be neglected or violated. I
haven't gone into any of that because I don't want the argument shifted to
whether my projections or other goals are acceptable excuses (to you), and
to have to fend off further attempts to get me to do what I have decided,
for reasons that satisfy me, not to do.

I have, up to now, been working on the presumption that we both have a
reference for understanding more about PCT.

You see how the pressure escalates? Now my refusal is being cast as
hypocrisy. But I know it isn't, so what I am controlling for remains
undisturbed.

when I first mentioned
the tool, you said it sounded like a very good approach;

I don't recall the "very", but it is a good approach that is worth
investigating. So investigate it, and tell me how it came out. But do it
with good data.

you claim
to be one of the very few people who are good practiced trackers, whose
results using the tool you would trust. You even ran a few new tracks to
illustrate the point.

No, people who are good enough trackers are not at all rare. It simply takes
time enough on the task for reorganization to be completed, after which the
control parameters become stable enough for a model with constant parameters
to fit the data well. If you continue to practiced at one task, you will see
the fit getting better and better until it reaches an asymptote, or gets
close enough to it that diminishing returns are a factor. This has nothing
to do with how well you control; some people, at asymptote, control better
than others. What it has to do with is stability of the parameters over a
one-minute run. While you're reorganizing, the parameters are changing and a
model with fixed parameters doesn't fit very well. There's no point in
drawing any conclusions about how well the model fits until reorganization
has essentially stopped.

I've discovered one of the sleep-study PCs that still has the software
on it, and isn't being used at the moment (it will be, quite soon).
On it, I have run a couple of tracks myself to see whether everything
still works properly.

I suggest running more than a "couple of tracks." If you keep running them
and recording the model-human correlations, you will see the fit getting
better and better, even if you don't approach super-skillful tracking. When
the fit stops getting better, THAT is the time to start analyzing the data.
Since a tracking run takes about one minute, you should be able to get in at
least 20 or 30 of them per hour, and in an hour or so you should be getting
much better fits.

I haven't yet tried to analyze the data, but it
seems likely to be possible. But I'm not a skilled, practiced tracker
on these tasks, as you are. Whatever my results, they will be subject
to the same criticisms you level at the sleep study subjects.

What good is a model that only fits the behavior of 3 people in the world?
You should be able to get good results with any reasonably normal person, if
you have the patience to provide the practice time and monitor the change in
fit until it has stabilized. I repeat, we are using a model with constant
parameters, and it can't be expected to fit the behavior of a control system
in which the parameters are continually changing. That would take a more
advanced model or a different method of analysis, which I can't now provide.

I know that your situation is difficult, but it seems likely that you will
not get the data you want until you take it yourself. That's very much the
situation all the other modelers are in, too.

Best,

Bill P.