[From Rick Marken (2017.03.27.1130)]
Powers1978.pdf (1.46 MB)
Leeanne Wright (2017.03.27.1.39 AEST)
Hi Rick, Martin & Bruce,
LW: Â I continue to be fascinated by this thread!Â The topic of significance-testing in psychology seems to be very salient at the present time. Therefore, good research fodder, perhaps!Â Maybe, the time has finally come ?Â
RM: At Martin&‘s behest I am continuing this discussion as a new thread with the subject head “Behavioral Illusion” even though I think that the behavioral illusion described in Powers’ 1978 Science article (which I have attached; the description of the behavioral illusion begins on p. 425) is directly relevant to significance testing that was being discussed in the “p-hacking” thread. The reason for the relevance is this: Significance testing is most often used in psychology to decide whether one can reject with a sufficiently small probability of being wrong (making a Type I error) the null hypothesis that the independent variable in an experiment did not have an effect on the dependent variable. When you reject the null hypothesis (with only a small probability of being wrong) you are deciding that the independent variable actually does have an effect on the dependent variable. The relevance of this to the behavioral illusion is that Powers shows that, when the systems under study are control systems (called N-systems in the paper), any apparent effect of an independent variable on a dependent variable is an illusion.Â
RM: The “p-hacking” thread noted some ways that significance testing can be abused. One way – the way that disturbed me most when I was a conventional psychologist – was to interpret the significance level as a measure of the probability of the null hypothesis being true. If the results are significant at the .001 level, that is taken to mean that null hypothesis more unlikely to be true than if the results are significant at the .01 level. Thus, the significance level is taken to be an inverse measure of how likely the alternative hypothesis is to be true. As Bruce A. so clearly pointed out, this is a very misleading way of reporting the results of a significance test; the significance level tells only the probability that your decision to reject the null hypothesis is a false alarm (a Type I error). Also, significance testing can be “hacked” in various ways, one being to just eliminate deviant data points and another being to keep increasing the sample size until your statistic is “significant” (with a large enough sample you can always get significance).Â
RM: But the behavioral illusions makes all the problems with significance testing moot. That’s because once you know that you are trying to understand the behavior of a living control system, the aim of your research is no longer to see if various independent variables (stimuli) have an effect on various dependent variables (responses). The aim of your research is to determine the variables that the system controls and how it controls them. The fundamental problem with significance testing, from a PCT perspective, is that is is based on a causal model of behavior; a model that says that the input to the system (the effect of the independent variable) is the ultimate cause of the output of that system (the dependent variable). That is, significance testing in psychology is based on the wrong model of the nature of living systems.
RM: So what I recommend for going forward on this “behavioral illusion” thread is that those of us interested in doing PCT research (which may be just you and me Leeanne;-) should read (or re-read) the attached 1978 Science paper by Powers and then discuss the implications of Powers’ analysis for research in psychology. Let’s see what’s going on in some experiments where the results were analyzed using significance tests and see how we might interpret those results in the context of the analysis in Powers 1978 Science paper. Then let’s propose some experiments consistent with the rather abstract proposals for research presented in that paper (as Experiments 5 and 6).Â
Richard S. MarkenÂ
"Perfection is achieved not when you have nothing more to add, but when you
have nothing left to take away.â?
Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â --Antoine de Saint-Exupery