The results indicate that the investigators contributing to Volume 61 of the Journal of Abnormal and Social Psychology had, on the average, a relatively (or even absolutely) poor chance of rejecting their major null hypotheses, unless the effect they sought was large. This surprising (and discouraging) finding needs some further consideration to be seen in full perspective.
First, it may be noted that with few exceptions, the 70 studies did have significant results. This may then suggest that perhaps the definitions of size of effect were too severe, or perhaps, accepting the definitions, one might seek to conclude that the investigators were operating under circumstances wherein the effects were actually large, hence their success. Perhaps, then, research in the abnormal-social area is not as “weak” as the above results suggest. But this argument rests on the implicit assumption that the research which is published is representative of the research undertaken in this area. It seems obvious that investigators are less likely to submit for publication unsuccessful than successful research, to say nothing of a similar editorial bias in accepting research for publication. Consider this paradigm: 100 investigations are undertaken in which, in fact, there is actually a medium population effect. From the above findings, about SO get positive results and are likely to come to publication; the other SO fail to reject their (assumed false) null hypotheses and are unlikely to come to publication. Thus, the general success of the articles in the volume under review does not successfully argue for their antecedent probabilities of success being any higher than the results of the analysis suggest, or, equivalently, that the criteria for size of effect used were overly stringent."