Testing for Control In Experiments (was Re: Control of and in imagination)

CSGnet Archive CSG2010
21

[From Bill Powers (2010.08.01.1530 MDT)]

Bruce Gregory (2010.08.01.1658 EDT) --

BG: It may be, of course, that the discovery of control, like the discovery of the mechanisms by which signals are transmitted in the brain, while significant in itself, does not lead to great advances in learning how behavior "really works." You can get pretty far by simply assuming that once neural networks become established, it takes energy to rewire them. The brain, it seems, tends to resist extensive rewiring (for perfectly good reasons).

At the moment I have no defenses against this sort of comment. I'm wondering what the point is of keeping on with PCT, when this is all it leads to. It's not your fault, you haven't said anything more discouraging than in the past. But I'm not back on local time yet and I don't seem to have any goals to beckon me to keep pushing back against the disturbances. I expect to get them back in a few days but for now, good luck with your new insight.

Bill P.

22

[From Bruce Gregory (2010.08.01.1826 EDT)]

[From Bill Powers (2010.08.01.1530 MDT)]

Bruce Gregory (2010.08.01.1658 EDT) --

BG: It may be, of course, that the discovery of control, like the discovery of the mechanisms by which signals are transmitted in the brain, while significant in itself, does not lead to great advances in learning how behavior "really works." You can get pretty far by simply assuming that once neural networks become established, it takes energy to rewire them. The brain, it seems, tends to resist extensive rewiring (for perfectly good reasons).

At the moment I have no defenses against this sort of comment. I'm wondering what the point is of keeping on with PCT, when this is all it leads to. It's not your fault, you haven't said anything more discouraging than in the past. But I'm not back on local time yet and I don't seem to have any goals to beckon me to keep pushing back against the disturbances. I expect to get them back in a few days but for now, good luck with your new insight.

BG: The best defense is a good offense. Pick a few recent studies in social psychology or behavioral economics. Demonstrate how the studies would have been improved had the experimenters understood PCT. Suggest what the experimenters might have learned that they did not learn from the designs they used. It can't be that hard, can it? If it is that hard, maybe we can learn something.

By the way, the insight is not new and it is not mine.

Bruce

23

[From Rick Marken (2010.08.01.2000)]

Bruce Gregory (2010.08.01.1826 EDT)

BG: The best defense is a good offense. Pick a few recent studies in
social psychology or behavioral economics. Demonstrate how the studies
would have been improved had the experimenters understood PCT.

This was already done, in general terms, by Bill in the 1978 Psych
Review article that is on the reading list I posted. To repeat it's

Powers, W. T. (1978). Quantitative analysis of purposive systems: Some
spadework at the foundations of scientific psychology. Psychological
Review, 85, 417�435.

I also have a paper that addresses this issue [Marken, R. S. (2005)
Optical Trajectories and the Informational Basis of Fly Ball Catching,
Journal of Experimental Psychology: Human Perception & Performance, 31
(3), 630 � 634] but it specifically addresses experiments on human
performance (catching fly balls, of course), not in social psychology
or behavioral economics.

If you would like to discuss specific experiments in social psychology
or behavioral economics, I think it would be best if you picked a
recent study in one of these areas that you think is important. If we
pick the study and show how it could have been improved had the
experimenters understood PCT you might think we were attacking a straw
man.

I thought Gavin was going to post an experiment in behavioral
economics. But he didn't. So maybe you could?

Best

Rick

···

--
Richard S. Marken PhD
rsmarken@gmail.com
www.mindreadings.com

24

[From Bruce Gregory (2010.08.02.0616 EDT)]

[From Rick Marken (2010.08.01.2000)]

If you would like to discuss specific experiments in social psychology
or behavioral economics, I think it would be best if you picked a
recent study in one of these areas that you think is important. If we
pick the study and show how it could have been improved had the
experimenters understood PCT you might think we were attacking a straw
man.

[From Bruce Gregory (2010.07.21.1238 EDT)]

Just to get the ball rolling, here is an experiment done by Dan Ariely’s group. A group of students were given instructions as to how to fold one of two origami figures. They were told they had to surrender the finished work to the experimenter. When they were done they were allowed to bid on their own works. Those who created the object tended to bid much more for it than student passerbys who had not folded the objects. When both groups of students were given the opportunity to bid on origami figures made by origami experts, the students to tended to place a similarly high value on the pieces (comparable to what the student folders bid on their own efforts).

How might this experiment have been performed differently if Ariely had been familiar with PCT? What might he have learned that he did not learn as a result of performing the experiment in the original way?

Bruce

25

[From Bill Powers (2010.08.02.0505 MDT)]

Bruce Gregory (2010.08.02.0616 EDT) --

Just to get the ball rolling, here is an experiment done by Dan Ariely's group. A group of students were given instructions as to how to fold one of two origami figures. They were told they had to surrender the finished work to the experimenter. When they were done they were allowed to bid on their own works. Those who created the object tended to bid much more for it than student passerbys who had not folded the objects. When both groups of students were given the opportunity to bid on origami figures made by origami experts, the students to tended to place a similarly high value on the pieces (comparable to what the student folders bid on their own efforts).

How might this experiment have been performed differently if Ariely had been familiar with PCT? What might he have learned that he did not learn as a result of performing the experiment in the original way?

He might have learned why some students bid more and others less for the same items. Of course that would have to be determined one student at a time rather than for "students." MOL-type interviews would reveal why some of the students bid less for their own work than some of the other students did, and why some passers-by bid more. I trust that the findings are the usual sort of psychological fact, which is not true of a substantial number of subjects. If not, I apologize for being pessimistic. Do you have the actual numbers? It would be interesting to see the breakdown of positive and negative instances of the group findings.

Best,

Bill P.

26

[From Bruce Gregory (2010.08.02.1202 EDT)]

[From Bill Powers (2010.08.02.0505 MDT)]

Bruce Gregory (2010.08.02.0616 EDT) --

Just to get the ball rolling, here is an experiment done by Dan Ariely's group. A group of students were given instructions as to how to fold one of two origami figures. They were told they had to surrender the finished work to the experimenter. When they were done they were allowed to bid on their own works. Those who created the object tended to bid much more for it than student passerbys who had not folded the objects. When both groups of students were given the opportunity to bid on origami figures made by origami experts, the students to tended to place a similarly high value on the pieces (comparable to what the student folders bid on their own efforts).

How might this experiment have been performed differently if Ariely had been familiar with PCT? What might he have learned that he did not learn as a result of performing the experiment in the original way?

He might have learned why some students bid more and others less for the same items. Of course that would have to be determined one student at a time rather than for "students." MOL-type interviews would reveal why some of the students bid less for their own work than some of the other students did, and why some passers-by bid more. I trust that the findings are the usual sort of psychological fact, which is not true of a substantial number of subjects. If not, I apologize for being pessimistic. Do you have the actual numbers? It would be interesting to see the breakdown of positive and negative instances of the group findings.

BG: You are suggesting an interesting extension of the experiment, but I was interested in how the experiment would have been conducted differently if Ariely had understood PCT. Perhaps the answer is that the experiment would not have been conducted at all if Ariely had understood PCT.

A PCT-savvy researcher might not be interested in such questions such as, how many people participate in retirement plans if they must opt out compared with if they must opt in? Perhaps some subjects are controlling the perception, "opt out" and others controlling the perception "opt in." Of course, if you are interested in increasing participation in retirement programs, it may not matter to you exactly which perception each individual is controlling.

If your goal is to discover which perception a particular individual is controlling, PCT is the way to go. I suspect that is why MOL is so attractive. PCT is the science of individual behavior, not group behavior. As far as I can tell, PCT is more about how than about why. Nothing wrong with that, but it has limited the impact of PCT. We can learn that a system is controlling a variable by observing the way the system counters disturbances. Why the system is controlling that particular variable with that particular reference level, not so much.

I am not sure what you would hope to learn from the breakdown of positive and negative instances in the group findings. Ariely apparently has yet to publish these results. IKEA effect Archives - Dan Ariely

Bruce

27

[Martin Taylor 2010.08.03.11.00]

Amazing! I'm away for a week and there are over 30 messages on CSGnet, wheras on other occasions a week or more passes without a single message!

Here I'm following up on ...

[Martin Taylor 2010.07.27.14.52]

[From Rick Marken (2010.07.26.2300)] Your model requires the addition of a model of sensory storage, or it requires the assumption that the interval can be chosen without using the pI-pA control system that both models have been assuming to produce the choice of interval. If your model can turn the sensory data (the original pI) into the choice of interval (your new pI) without using a relationship control unit, then your whole structure becomes the S-R structure that for several months you said my model was.

Howsoever that may be, are you still in agreement with yourself that both
models involve a relationship control system that has inputs pI and (real or
imagined) pA (i.e. is controlling either pI-pA (yours) or pi-iA (mine)), and
that both will reliably produce an overt response that corresponds closely
to the interval in which the subject perceived the signal to have occurred?

Yes.

I'll take that as a "Yes".

Much follows from that "Yes". Thank you.

Martin

In continuing this thread, Rick has not addressed in his model the workings of the sensory memory system that would be needed in my proposed experiments to make his "relationship between perceived stimulus and actual response" model work. I conclude that at this point he has no model for the control systems involved in those experiments. All he has is a reversion to name-calling, in labelling my model as "open-loop", which it most definitely is not.

None of that matters, since the only point of disagreement is the circuitry that allows a person to answer a question about a state of the environment. It's a worthy topic, and one worth pursuing, since it is much more generally applicable than just in a psychophysical experiment. However, the real topic of this entire succession of renamed threads that goes back to February in this incarnation is whether psychophysical or other studies can tell anything about the properties of systems internal to the brain.

What follows from Rick's "Yes" is that they can indeed, because regardless of which (if either) model is correct of how a person answers questions about the perceived state of the environment, if the overt response in the 4AFC experiment reliably tracks the interval in which the subject has perceived the signal to have occurred, then any errors must be attributed to the pathway between the sensory systems and the relationship control unit that matches the answer to the interval in which the signal was actually presented. It is that pathway that is the subject of the experiment, and its accuracy under a variety of environmental conditions allows for many of its properties to be determined. In audition, properties such as acceptance and rejection bandwidth, internal noise levels, interactions between signals of different types and in different spatial locations, and so forth, can be (and have been) adduced.

Of course, as I discussed earlier in my description of how good experimental practice in psychophysics actually (unknown to most psychophysicists) requires testing for the controlled variable, one has to be sure that the subject actually is controlling for (in the N-AFC case) providing an overt response that corresponds to the interval containing the signal, but given that this test is satisfactorily applied, the rest follows.

In other words, conventional psychophysics experiments using good experimental practice do provide an effective window into the workings of some aspects of perceptual function.

Martin

28

[Martin Taylor 2010.08.03.1123]

[From Bill Powers (2010.08.01.0855 MDT)]

I don't have anything in particular against the topics that have been discussed in the last month or so on CSGnet, but they aren't going in directions that interest me and I have little to contribute to them that would satisfy any of the current discussants. Applying PCT-like terminology can be useful, but if the underlying ideas are still the same old ones that have been around for 50 to 200 years it's not likely that anything new will be added to PCT as a result.

Is that the objective? I thought the primary objective was to develop PCT so that its insights illuminate the actual behaviour of people, and put its study on a rigorous scientific foundation, rather as the Mendeleev periodic table of the elements made sense of a lot of chemistry that had previously just been unrelated observations.

Personally, I'm not interested in simply "adding something new" to PCT, although it is interesting and probably useful when something new is in fact added.

An old manuscript showed up while I was looking for something else; it was the first draft of what eventually became Making Sense of Behavior. The working title was "Starting Over." That's still what I want to do, though it's getting a little late in the day. I want to find out if there are really levels of control, and see the research started that will begin to establish their nature. I want to know if apparent instances of "up-side-down" levels are real -- that is, wanting to teach algebra to make enough money to eat. Could there be some organization other than a hierarchy? I want to know how reference signals held in memory drift with time. I want to know how we reorganize in new environments to perceive their properties and learn to control them. I want to know if there is really an organized environment Out There, and if so, whether the kinds of things we perceive are linked in any systematic way to properties of reality. I (like Richard Kennaway) want to build PCT robots, or see them getting built; I want to help develop a curriculum for teaching the basics on which PCT rests.

There is a lifetime of projects concerning living control systems waiting for someone to get interested in them. Most of the old problems will eventually fade away as people lose interest in them, and PCT will introduce new ones that more directly concern the question of how behavior really works, how it is really organized, what we can really do about human difficulties.

This is inspiring. Thanks for writing it.

Martin

29

[From Rick Marken (2010.08.03.1050)]

�Martin Taylor (2010.08.03.11.00)--

In continuing this thread, Rick has not addressed in his model the
workings of the sensory memory system that would be needed in my
proposed experiments to make his "relationship between perceived
stimulus and actual response" model work.

Actually, I did address it. I said it's it's irrelevant to testing the
difference between our models. You proposed an experiment and then
claimed that my model couldn't handle it because it needed a sensory
memory system to make it work. I explained that this is not the way I
do science. My approach to science involves designing experiments that
will test the difference between existing models. My proposed
experiment does that. Your proposed experiment doesn't. If the subject
in my experiment does not change their answer when the signal interval
changes during a trial then your model would be supported and mine
rejected; if the subject immediately changes their answer when the
signal interval changes during a trial then my model would be
supported and yours rejected.

I conclude that at this
point he has no model for the control systems involved in those
experiments.

So? What does my not having a model (yet) for some experiment you came
up with have to do with the relative merits of our models of the
experiment we've been talking about?

All he has is a reversion to name-calling, in labelling my
model as "open-loop", which it most definitely is not.

Your model does seem open loop with respect to the perceived
relationship between signal interval and answer. But it doesn't matter
what I call it; it matters how the model behaves. What we need is an
experiment to see whether people behave the way your model says they
do. My experiment tests that.

In other words, conventional psychophysics experiments using good
experimental practice do provide an effective window into the workings of
some aspects of perceptual function.

I would rather do the experiment first to see if that conclusion
(based on your model) is warranted. I still believe that my model of
the behavior in psychophysical experiments is better than yours and
that, therefore, the PCT caveats about the behavioral illusion hold
for the results of all these experiments.

Best

Rick

···

--
Richard S. Marken PhD
rsmarken@gmail.com
www.mindreadings.com

30

[From Bruce Gregory (2010.08.03.1446)]

[From Rick Marken (2010.08.03.1050)]

Martin Taylor (2010.08.03.11.00)--

Boys! Boys! Play nice. If two of the very few people who claim to understand PCT cannot agree on what constitutes a valid experiment, what hope is there for the rest of us?

No wonder I'm incompetent! All the world, save me and thee, and I have my doubts about thee...

Bruce

31

[From Bruce Gregory (2010.08.03.1459)]

[From Rick Marken (2010.08.03.1050)]

Martin Taylor (2010.08.03.11.00)--

Boys! Boys! Play nice. If two of the very few people who claim to understand PCT cannot agree on what constitutes a valid experiment, what hope is there for the rest of us?

No wonder I'm incompetent! All the world, save me and thee, and I have my doubts about thee...

Bruce

32

[Martin Taylor 2010.08.03.15.10]

[From Rick Marken (2010.08.03.1050)]

  Martin Taylor (2010.08.03.11.00)--
In continuing this thread, Rick has not addressed in his model the
workings of the sensory memory system that would be needed in my
proposed experiments to make his "relationship between perceived
stimulus and actual response" model work.

Actually, I did address it. I said it's it's irrelevant to testing the
difference between our models. You proposed an experiment and then
claimed that my model couldn't handle it because it needed a sensory
memory system to make it work.

True, it does. When you include that in some model, we could conceivably compare them.

  I explained that this is not the way I
do science. My approach to science involves designing experiments that
will test the difference between existing models. My proposed
experiment does that. Your proposed experiment doesn't.

We live in very different worlds, it seems. I'm not at all clear how to get out of this "does" "does not" cycle. What is simply a basic observation to me is fantasy to you, and vice-versa.

  If the subject
in my experiment does not change their answer when the signal interval
changes during a trial then your model would be supported and mine
rejected; if the subject immediately changes their answer when the
signal interval changes during a trial then my model would be
supported and yours rejected.

Two problems: (1) The models would behave the same in your proposed experiment, and (2) changing the interval (the spatial location) of the signal during its presentation would greatly change the signal's detectability. Even if you disbelieve ALL the studies and theoretical analyses that support (2), even the possibity that is is true makes your proposed experiment invalid.

I conclude that at this
point he has no model for the control systems involved in those
experiments.

So? What does my not having a model (yet) for some experiment you came
up with have to do with the relative merits of our models of the
experiment we've been talking about?

"The experiment we have been talking about" doesn't exist. We (or at least I) have been trying to suggest an experiment that would distinguish between whether the subject in a N-AFC detection experiment controls (a) a relationship between the perception of the interval that contains a signal and a perception of the numerical name of that interval or (b) a relationship between the perception of the interval and a perception of physically pushing a button or voicing a numeral or waving an appropriate finger or ... (any suitable physical behaviour that the experimenter could reliably interpret as representing a numeral).

I produced a pair of experiments that in different ways allow these to be distinguished. You produced an experiment that changes the "stimulus" in the experiment, altering what the subject has to respond about, and an experiment, moreover, in which the two models would behave identically (as I pointed out in an earlier message).

...I still believe that my model of
the behavior in psychophysical experiments is better than yours and
that, therefore, the PCT caveats about the behavioral illusion hold
for the results of all these experiments.

Since there is no possibility for the subject's actions to influence the "stimulus" presentation, it might be instructive if you avoided or explained how the cliche "behavioural illusion" could apply in this case.

Martin

33

[From Rick Marken (2010.08.03.1350)]

�Martin Taylor (2010.08.03.15.10)--

Rick Marken (2010.08.03.1050)--

RM: My approach to science involves designing experiments that
will test the difference between existing models. My proposed
experiment does that. Your proposed experiment doesn't.

MT: We live in very different worlds, it seems.

I'm afraid that this is the problem. We just have completely different
ideas of how to go about doing science. I don't know if it is
resolvable.

RM: �If the subject
in my experiment does not change their answer when the signal interval
changes during a trial then your model would be supported and mine
rejected; if the subject immediately changes their answer when the
signal interval changes during a trial then my model would be
supported and yours rejected.

MT: Two problems: (1) The models would behave the same in your
proposed experiment

If this is true then how about proposing a variant of the experiment
that would distinguish the models -- the existing models, not your
model and some imagined new model of mine.

and (2) changing the interval (the spatial location) of the
signal during its presentation would greatly change the signal's
detectability.

So what? The workings of our models don't depend on detectability;
detectability affects only how well the subject controls (a
perception, in my model, an imagination in yours). Indeed, in my
version of the experiment there would be no detectability issue at
all. I would present a dot that is clearly visible in one of N spatial
locations; the position of the dot would (or would not) change during
the trial. Our models would behave the same in this experiment as they
do in the N-AFC experiment, except that control would be better. In
your model, the (now always correct) answer would be made in
imagination and then the reference for the overt answer would be
routed to a lower level system; in my model the answer would be made
as an overt response which is part of the controlled perception; the
relationship between perceived signal interval and overt answer.

RM: So? What does my not having a model (yet) for some experiment
you came up with have to do with the relative merits of our models of the
experiment we've been talking about?

MT: "The experiment we have been talking about" doesn't exist.

I thought we were talking about models of behavior in an N-AFC experiment.

MT: We (or at least I) have been trying to suggest an experiment that
would distinguish between whether the subject in a N-AFC detection
experiment controls (a) a relationship between the perception of the
interval that contains a signal and a perception of the numerical name
of that interval or (b) a relationship between the perception of the interval
and a perception of physically pushing a button or voicing a numeral or
waving an appropriate finger or ... (any suitable physical behaviour that
the experimenter could reliably interpret as representing a numeral).

Yes, we are trying to design an experiment that would distinguish
between two models. But both the models (a and b) that you describe
here sound like versions of my model. Your model controlled the
relationship between interval and answer only in imagination.

I produced a pair of experiments that in different ways allow these to be
distinguished.

If a) and b) are actually descriptions of the two different models,
then why say that my model has to be changed to account for the
results? This is where our different understandings of science create
what might be a deadlock. For me, the way to compare two different
models is to design a test where the two models make different
predictions, as in Eddington's test, during a solar eclipse, of
whether light from a distant star is bent as it passes the sun (as per
relativity) or not (as per Newton).

You produced an experiment that changes the "stimulus"
in the experiment, altering what the subject has to respond about, and an
experiment, moreover, in which the two models would behave identically
(as I pointed out in an earlier message).

Of course you have to make changes to an experiment in order to test a
theory. But the changes you make should be ones that result in
different predictions by the theories. Your experiment makes a lot of
changes but they don't test the difference between our theories as
they exist (at least according to you). That's like proposing a test
to compare relativity to Newton's model of gravity that requires that
one of the theories be changed in order to account for the not yet
obtained results. If that approach makes sense to you then I can see
no way for us to resolve our difference.

Since there is no possibility for the subject's actions to influence the
"stimulus" presentation, it might be instructive if you avoided or explained
how the cliche "behavioural illusion" could apply in this case.

Because the subject's actions affect a controlled variable (which I
hypothesize to be the perceived relationship between actions and
stimulus) which is also affected by the stimulus (which is an
independent variable).

Best

Rick

···

--
Richard S. Marken PhD
rsmarken@gmail.com
www.mindreadings.com

34

[Martin Taylor 2010.08.04.11.22]

[From Rick Marken (2010.08.03.1350)]

  Martin Taylor (2010.08.03.15.10)--

Rick Marken (2010.08.03.1050)--
RM: My approach to science involves designing experiments that
will test the difference between existing models. My proposed
experiment does that. Your proposed experiment doesn't.

MT: We live in very different worlds, it seems.

I'm afraid that this is the problem. We just have completely different
ideas of how to go about doing science. I don't know if it is
resolvable.

I have no evidence that what you say might be true. You carefully omitted to quote the relevant part of the quote. I had a quite different intention for "we live in very different worlds". Here is an example:

MT: We (or at least I) have been trying to suggest an experiment that
would distinguish between whether the subject in a N-AFC detection
experiment controls (a) a relationship between the perception of the
interval that contains a signal and a perception of the numerical name
of that interval or (b) a relationship between the perception of the interval
and a perception of physically pushing a button or voicing a numeral or
waving an appropriate finger or ... (any suitable physical behaviour that
  the experimenter could reliably interpret as representing a numeral).

[RM] Yes, we are trying to design an experiment that would distinguish
between two models. But both the models (a and b) that you describe
here sound like versions of my model. Your model controlled the
relationship between interval and answer only in imagination.

[MT now] In my world, a number is qualitatively different from a button press, a vocal utterance, or a waving finger. In yours, they are all the same. In my world, one can enumerate objects, one can perceive, say, how many times a button is pressed, but the number is NOT the button press. One can imagine a number, one can imagine a button press, but those imaginings are very different. I can't say "button press times button press equals waggle this finger" and make any sense, but I can say "two times two equals four" and have almost everyone in the world understand me. Our worlds are very different.

RM: If the subject
in my experiment does not change their answer when the signal interval
changes during a trial then your model would be supported and mine
rejected; if the subject immediately changes their answer when the
signal interval changes during a trial then my model would be
supported and yours rejected.

MT: Two problems: (1) The models would behave the same in your
proposed experiment

If this is true then how about proposing a variant of the experiment
that would distinguish the models -- the existing models, not your
model and some imagined new model of mine.

I did that, in two quite different variants. You rejected them for reasons that are totally obscure to me, just saying "they wont distinguish the models". As for an "imagined new model" of yours, I have to imagine it because you haven't yet described it.

  and (2) changing the interval (the spatial location) of the
signal during its presentation would greatly change the signal's
detectability.

So what? The workings of our models don't depend on detectability;
detectability affects only how well the subject controls (a
perception, in my model, an imagination in yours).

Again, we live in different worlds. In my world, when one is trying to determine the properties of a perceptual system, one works with the perceptual system in question. We are trying to determine whether a classical psychophysical experiment correctly assesses the ability of a person to detect a signal under different presentation conditions. You accepted that both our models would permit that, which should have ended the discussion that started years ago and resumed in February.

You may remember that I did note [Martin Taylor 2010.08.03.11.00] "...the only point of disagreement is the circuitry that allows a person to answer a question about a state of the environment. It's a worthy topic, and one worth pursuing, since it is much more generally applicable than just in a psychophysical experiment."

If you want to pursue that topic, I suggest changing the subject line to "Answering questions about perceptions" or something similar. That's where our models differ -- not in their ability to provide answers, but in their methods for achieving this result.

Martin

···

On 2010/08/3 4:48 PM, Richard Marken wrote: