[from Joel Judd]
Martin, Rick, Bill, etc.
Here's my take on the stat thing having looked through literature in my
field (SLA).
I think Bill gives too much credit to editors and other "mentors."
Replications are not published becasue thay are not wanted, in many
instances. Graduate training includes spoken or unspoken directives to
finally "go where no man has gone before." A replication does not provide
committees with evidence that a student is original and inventive, it shows
an advisor's lack of progressive influence on the advisee.
But can a replication (of a study using group statistical data) even be
carried out? Think about it. Taylor (1958) and Runkel (1990) have. I look
at the studies in SLA, and I don't know if even 15% of them could truly be
replicated. No one has true random samples. This is the *basis* for
methods of relative frequency. Nobody has them!! How can I replicate a
non-random sample? Maybe it's actually easier--I can use anyone I want!
Runkel goes even further: Suppose I want to replicate a random sample of a
population. Can I repeat a study carried out at time X six months later?
Six years later? Can I claim anything for a true random sample when I take
eight months to complete the experiment--haven't the population
characteristics changed by then? But these are moot points for me, I
repeat, no one (of widespread notoriety) uses random samples in studying L2
acquisition.
The point of studied examinations of social science research and group
statistics (like the two references listed above) is: you're using the
wrong tool to find out about individual functioning. Relative frequencies
were never intended to be used to uncover explanations (i.e. theories) of
human functioning. By their nature, they CANNOT DO IT. The facts (at
least in my field) are that there is widespread looking the other way as
the methods are misused and misinterpreted, that shoddy results become SLA
fact over time, and that ever more powerful and hopelessly uninterpretable
techniques are applied to language learners each year. These should be
clues that something is amiss.