Testing with models

[From Rick Marken (960617.2045)]

Me re: Peter Burke's experiment

The Test for the Controlled variable is done by seeing how well the model
fits the data.

Jeff Vancouver 960617.15:30]

Excuse me! What is the controlled variable? Fit?

No. Of course not.

The _hypothesized_ controlled variables are the variables controlled by
the model. In Burke's model there were three controlled variables that
I described in my post: 1) participation in the exchanges 2) the actual
number of points received and 3) the time taken to accept the offer.

A good example of using modelling to Test for controlled variables is
described in Powers "A Feedback Model of Behavior" (Chapter 3 in LCS). Here
Bill applied a control model to determine the variable controlled by rats
in a shock avoidance experiment. Bill tested for the controlled variable by
trying two different definitions of the variable controlled by the model:
rate of shock and probability of shock. The model that fit best was the one
controlling rate of shock -- suggesting that the variable controlled by
the rats is rate rather than probability of shock. I suggest that you
what Peter tried to do in his study, which was to determine what an
organism is controlling by matching a control model to the organism's
behavior. The best guess at the controlled variable is the perceptual
function that let's the model perform most like the organism.

Testing for controlled variables with a model is usually _ not_ the best
way to determine what an organism is controlling; it was the only approach
Bill could use with the rats because the data had already been collected.
A far more efficient and accurate procedure would have been to do the Test
for the controlled variable in the usual way; by introducing disturbances
that should have a pronounced effect on a variable if it is _not_ controlled.
This allows you to hone in on better and better definitions of the
controlled variable. Once you have very precise data -- clear-cut
control -- you can compare the data to a control model; since you already
know the variable that should be controlled by the model, what you will
learn from the model is something about the organism's gain and dynamics.

Where is the word (or concept of) model in the Test?

In the concept of a controlled _perceptual_ variable. The control model
controls a representation (perception) of some state of affairs in
the world; when you find evidence that some function of environmental
events that is being protected from disturbance you have, in fact, tested
a model that says organisms are controlling their perceptions.

It seems that Burke a) guessed at some controlled variables, b) set
up a situation that disturbed those variables, and c) measured the
state of the variables from an external perspective.

Peter didn't do the test this way, probably becuase there were several
possible controlled variables involved. Also, it may have been difficult
to calculate the expected value of the hypothesized controlled variables
under the assumption of no control; so there was no frame of reference
against which to judge whether the variables were controlled or not. The
model actually provides a frame of reference for evaluating whether
control is happening; in this case the model shows how the hypothetical
controlled variables would behave if the WERE (rather than were not)
unded control.

The [ecological validity] criticism is not that the findings are
worthless (note, I would suggest it be a discussion point), but that one
be sure to fully appreciate the limitations in the data.

OK. I'll accept that.

Best

Rick

[from Jeff Vancouver 960620.12:35]

[From Rick Marken (960617.2045)]

Me re: Peter Burke's experiment

>The Test for the Controlled variable is done by seeing how well the model
>fits the data.

Jeff Vancouver 960617.15:30]

>Excuse me! What is the controlled variable? Fit?

No. Of course not.

Phew, my world almost colapsed there.

A good example of using modelling to Test for controlled variables is
described in Powers "A Feedback Model of Behavior" (Chapter 3 in LCS). Here
Bill applied a control model to determine the variable controlled by rats
in a shock avoidance experiment. Bill tested for the controlled variable by
trying two different definitions of the variable controlled by the model:
rate of shock and probability of shock. The model that fit best was the one
controlling rate of shock -- suggesting that the variable controlled by
the rats is rate rather than probability of shock. I suggest that you
what Peter tried to do in his study, which was to determine what an
organism is controlling by matching a control model to the organism's
behavior. The best guess at the controlled variable is the perceptual
function that let's the model perform most like the organism.

Testing for controlled variables with a model is usually _ not_ the best
way to determine what an organism is controlling; it was the only approach
Bill could use with the rats because the data had already been collected.
A far more efficient and accurate procedure would have been to do the Test
for the controlled variable in the usual way; by introducing disturbances

Now I understand. There is the TEST the "usual way" and than there are
other ways.

The other way that is demonstrated by Burke and Powers is that of
comparing a model's behavior with subjects' behavior, where the model is a
control system model. Fine, welcome to cognitive psychology. Only the
nature of the model is not always a control system model in cognitive
psychology. Perhaps we should not confuse the method with the content of
the theory being tested.

But I just want to make clear what has been established (so that Rick has
something to argue against). Sometimes, when exigencies (like archival
data) or ethics (like not being able to disturb certain variables)
prevent us from using the TEST, we may then use the test.

Given that Rick's version of the test (comparing results to a control
theory model) assumes the model, neither the TEST nor the test can be used
to test the theory. So, there must be more methods available as Rick
wishes to test the theory. Stay tuned?

ME:

>The [ecological validity] criticism is not that the findings are
>worthless (note, I would suggest it be a discussion point), but that one
>be sure to fully appreciate the limitations in the data.

OK. I'll accept that.

You are teasing me - I can change Rick's mind?

Later

JEff