[From Bill Powers (940927.0230 MDT)]
Bruce Nevin (940926.1457) --
I recognized that I have nothing to contribute but speculative theory
building until I am in a position to test for controlled variables and
to model the control of those variables. So that is an aim. That is,
I am controlling the perception of myself being in such a position, and
although my control of this perception is very poor at present I expect
it to improve with practice and persistence.
First, I hope you represent the position of a lot of listeners. Greg
Williams is right (I haven't seen Ed Ford's post on this subject yet --
our email has become extremely slow!). We need to get back to model-
building and testing. It may be just as informative to newcomers to see
how this is done as to discuss doing it in the abstract. Now all we have
to do is figure out how to do it ourselves.
Second, a made-up PCT Zen parable:
The apprentice observed the master struggling with a bottle and a cork,
knocking things off the table and muttering to himself.
"Master," said the apprentice, "I perceive that you are experiencing
some kind of error signal. If you will tell me what variable you're
controlling and your reference level for it, I will endeavor to augment
your loop gain or supplement your output function."
The master replied "Hang on to this bottle, will you? I'm trying to get
the goddamn cork into it."
ยทยทยท
--------------------------------------
RE: Publishing PCT experiments: ideas for discussion.
As long-time listeners will know, I preach doing experiments again and
again, modifying the model to improve its match to behavior each time
and reduce the unexplained variance, with the aim of having the
prediction error be no more than about 10% of the range of the variables
(which will get you correlations in the high nineties). And of course,
as much less error as possible. The ultimate aim is for the model to
predict within errors of measurement. If you have to simplify your
experiment to get such results, so much the better.
One way to help get this done is to recognize two classes of reports on
experiments: exploratory and completed. An exploratory experiment might
show a correlation between prediction and real behavior of 0.5 or even
less. When such an experiment is done, it should be published on CSG-L
with full details of procedure, conditions, hypothesis, etc. so others
can understand what you did and what results you got. Then others can
suggest ways of modifying the experiment or the hypothesis that will
reduce the unexplained variance, and another exploratory experiment can
be done and published. This process can be repeated until an acceptable
correlation is found, meaning that diminishing returns have set in.
Then the exploratory work can be summarized with the final model and
results for publication in Closed Loop.
Should we set a minimum correlation of prediction against experiment for
"completed" work? We could start low, say 0.85, and raise (or lower) the
ante until we get just enough papers to fill the journal. Or maybe we
could select experimental articles for publication in order of
predictive accuracy, for otherwise acceptable papers. A little selection
pressure?
Of course anyone can do this without putting the exploratory work on
CSG-L for discussion, but by publishing the exploratory results, you get
the advantage of many minds looking for ways to improve the experiment.
If you publish any computer code (or make it available through Bill
Silvert's file server or by direct email on request), or transmit any
instructions and methods used if there isn't any code, others can try to
replicate your experiment. This will help them understand what you're
trying to do, and will help them come up with useful suggestions.
My reason for suggesting this is to encourage people to try PCT
experiments even when they're unsure of how to proceed and haven't had
any luck in getting good results right away. It's also to try to
establish a professional milieu in which is is expected that final
published results in Closed Loop will represent an iterative process of
improving theory and experiment, rather than simply the outcome of a
one-shot test of a single hypothesis.
Needless to say, papers that never get to the "completed" stage will, by
custom, not be cited and the results will not be used in extended
reasoning.
One nice thing about PCT research is that you don't need a lot of
subjects for exploratory work; often one is enough to get started. If
another person tries the same experiment, you'll get n = 2, and so on.
So nobody has to go to great expense to do exploratory work, especially
if you keep it simple enough for others to try easily. When you've got
the bugs out, you can spend most of the grant money on testing many
subjects.
I'm also thinking that for publication of a final-results paper in
Closed Loop, we should have an absolute requirement that experimental
results be replicated by at least one person other than an author of the
article, preferably a person at a different institution. If nobody else
can varify your results, they're not worth publishing. If you can't get
at least one other person interested enough to do a replication of your
results, they're not worth publishing. This will not only improve the
results, it will keep the nut cases out, and will encourage people to
replicate other people's work, because if you're not willing to do
replications, who will replicate your work so _you_ can publish in CL?
This requirement for replication will, of course, rule out 1000-subject
experiments. That is good. Any hypothesis that needs 1000 people to test
it is probably wrong or trivial anyway. When you apply for a grant,
automatically apply for enough to fund an attempt at replication by
someone else. That's another way to encourage replication.
Another thought: every experimental paper offered for publication should
include the raw experimental data, or that data should be available
without restriction by email or on request by anyone willing to pay the
cost of a disk and mailer. This should be a rigid prerequisite for
publication in Closed Loop, and data, theory, and methods should be put
in the public domain. Let us not repeat the mistake of the biologists to
whom exploration of human biochemistry has become just a means of making
money. Any patented or copyrighted scientific information, or
information carrying any other restriction on its use, should be grounds
for rejection of a paper for publication in Closed Loop.
A neighbor of a relative of mine is a biochemist. He has said to me that
his goal is to find something really valuable so he can retire by the
age of 35. I think he's recently had to raise that to 40. I hope he
never makes it; his goal is clearly to stop doing biochemistry. Who
could ever trust any "finding" he came up with?
Under "exploratory experiments" I would include simulations without test
against human or animal (or plant) data. Here the point is to
demonstrate that your model will actually behave as you claim it will. I
would go so far as to say that any "completed" general theoretical
proposition must include a simulation or some proof that the proposition
will actually work. This would protect us against, for example, the
block diagrams found in personality research, most of which probably
will not work.
I would not include pencil-and-paper mathematics, because without
simulations it is impossible to tell whether what works with analytical
methods will work with a real, or even a simulated, system. I know that
one will get objections for sure, but I lay the suggestion out for
discussion.
---------------------------------------------------------------------
Best,
Bill P.