[From Bruce Abbott (971123.0855 EST)]
Phil Runkel on 22 Nov 97:
Thanks, Bruce, for the reply. I can only say that I find no comfort in
the notion of generalization as it is used by the devotees of casting
nets. I see no advantage in saying that some condition or effect is
general over 100 people when you have observed it to occur in only 55 of
them. Or 35 of them.
I agree. Does this surprise you? Then you have missed my point.
And WTP and I were only saying that when you see that 55 of 100 do
something, and then report that "They are all like that," or "They all
tend to be like that," then you are using exactly the same logic as when
you say, "They all like watermelon."
I agree again. Such statements are terrible. My point was not that such
statements are defensible. My point was that when doing research to
identify variables that have strong influences on individual behavior, the
finding that 55 of 100 show an effect, or even 35, is worth noting and
following up on.
A few years ago I assisted in a study that involved injecting an extremely
tiny amount of a drug directly into the main ventricle of a chick's brain.
The object was to assess the effect of the drug on isolation-induced
distress vocalization (DV), an index of what is termed "separation anxiety."
The chicks were administered their injection and then placed in a small (1
ft square) illuminated box, where their DVs were counted over a 5-min
observation period. To determine the dose response, different groups of
chicks received different doses, including 0 micrograms (drug vehicle only).
The average number of DVs observed varied in a highly systematic way with
the dose, but there was an occasional exception within some drug groups (an
animal whose DVs were in the same range as untreated animals). Dye tests of
the injection technique revealed that these exceptions were probably due to
the tip of the needle sometimes failing to enter the ventricle on rare
occasion, so that the drug was unable to diffuse rapidly to the target
structures in the brain. A highly reasonable inference is that the
exceptions seen in testing were probably examples of this error. But even
if they represented exceptions to the general effect of the drug, the study
revealed that the drug has a powerful effect on distress vocalizations in
(most) chicks.
Nor does it matter much how the group-based effect appeared, i.e., whether,
in each chick, there was a dose-dependent graded effect or whether the
effect of the drug was all-or-none with thresholds varying across chicks.
This was a preliminary screening to determine whether any effect could be
discerned. Once an effect was detected, the details could be determined
using other designs (including examining the effect in single subjects.)
My point was and is that such designs can reveal effects that are consistent
enough across individuals to produce reliable group differences, not that
the trends revealed by group means will necessarily apply to every (or even
any) individual. Marken's spreadsheet demos (both simulations) showed that
this assertion is correct, although he perversely continues to proclaim that
they disprove it.
If anyone would care to look back on my original post that started this
whole discussion, he or she would see that I made only modest claims for
what can be learned from such designs. My only concern was that, in
pointing out how results of studies using such designs have been misused
(e.g., claiming that the results apply to every individual), the strong
impression was being left that such designs have no valid uses at all. One
should not conclude that, just because a finding was obtained with a group
design, it has no validity at all. This claim is not defensible.
Regards,
Bruce