[From Bill Powers (2002.05.31.1007 MDT)]
Bruce Abbott (2002.05.30.1935 EST)--
Although I have no quarrel with your suggestion, I believe that a couple of
points need to be emphasized in this particular case. First, I cannot yet
classify the pill-bug experiment as a replication failure, as I do not know
the details of the procedure as given by Mueller. It is still very
possible that I did not arrange the required conditions. (I'll know only
after I have Mueller's actual description in my hands, competently
translated.)
Granted.
Second, Mueller was a physiologist, not a psychologist. If
the results prove either possible or impossible to replicate, then let's at
least assign credit or blame, if that's what we're doing, to the right
discipline.
I should have said "behavioral science," since that's what he was doing.
Third, I have personally conducted a number of replications
of published studies in my own area of work that reproduced the reported
results.
You have also, to my knowledge, tried to replicate another number of
published studies in your own area and found them unreproducible. But
actually I would expect the basic observations of EAB (if not the fancier
variations) to be fairly reproducible. My net was cast over a wider area,
of which EAB is only a small part. I'm not the first to have raised the
question of replication and reproducibility in psychology or behavioral
science or the social sciences, though I can't point to a source at the
moment.
In fact, in those areas it is typical to begin a new study by
including replications of previous manipulations, simply to be sure that
the phenomenon being investigated can be reproduced before going on to
investigate additional manipulations. With a little searching, you would
find much more replicating going on than you now believe is being done.
That could be, but what I have seen of replications has always shown that
the replicator couldn't resist adding some "little" twist of his own, or
making do with some different equipment, or different method, or different
subject population, because the original one was too costly or was
unobtainable or no longer existed. I think there's a general distaste for
doing literal replications, because it's hard enough to get money and time
for original research -- who wants to spend half a year simply doing
exactly what someone else has already done, changing nothing? Of course the
answer is, "Any real scientist." We can be confident about facts only if
they can be reproduced, the more exactly the better.
Also, as you and I discovered, simply repeating what another experimenter
did is far from easy. Many important details are left out of published
descriptions, and the only way to discover them is to get in touch with the
original experimenter -- who may not remember the details (correctly or at
all) and may no longer have the raw data. You know who I'm talking about. I
don't think I have ever seen a description of apparatus or methods in the
psychological literature -- including EAB -- from which I could reconstruct
the original experiment and be confident I was doing so. People can
interpret the term "replicate" very loosely.
Best,
Bill P.