Too Good to Be True: Publication Bias in Two Prominent Studies from Experimental Psychology
⚡ Contested ↗📌 Appears in:
Plain English Summary
Clever statistical detective work targeting Daryl Bem's famous 2011 precognition experiments. Francis asks a simple question: Bem reported positive results in 9 out of 10 studies, but given how small the effects were, should that many have actually worked? The expected hit rate was only about 6 out of 10 — meaning Bem's near-perfect record is itself suspiciously unlikely. Francis finds the same too-good-to-be-true pattern in unrelated verbal overshadowing research. The diagnosis: publication bias — the tendency to publish wins and bury losses — likely contaminated both literatures, making them unreliable as evidence. He suggests Bayesian analysis as a healthier alternative.
Research Notes
Key skeptical contribution to the Bem Feeling the Future controversy. Demonstrates that the seemingly impressive replication rate across Bem's experiments is itself statistically implausible, providing a quantitative basis for the suspicion that selective reporting inflated the evidence for precognition.
Applying the Ioannidis and Trikalinos (2007) test for excess significance to Bem's (2011) ten psi experiments and a set of verbal overshadowing studies, this analysis finds that the observed number of null hypothesis rejections substantially exceeds what would be expected given the experiments' statistical power. Bem's studies yield a pooled effect size of g* = 0.186, predicting 6.27 rejections out of 10, yet 9 were reported (p = .058). The verbal overshadowing literature shows a similar pattern (p = .022). These results indicate publication bias contaminates both literatures, rendering them uninformative as scientific evidence. Bayesian data analysis is proposed as a partial remedy.
Links
Related Papers
Cites
- Why Psychologists Must Change the Way They Analyze Their Data: The Case of Psi — Wagenmakers, Eric-Jan (2011)
- Must Psychologists Change the Way They Analyze Their Data? — Bem, Daryl J (2011)
- A Bayes Factor Meta-Analysis of Bem's ESP Claim — Rouder, Jeffrey N (2011)
- False-Positive Psychology: Undisclosed Flexibility in Data Collection and Analysis Allows Presenting Anything as Significant — Simmons, Joseph P (2011)
- A Practical Solution to the Pervasive Problems of p Values — Wagenmakers, Eric-Jan (2007)
Companion
- Correcting the Past: Failures to Replicate Psi — Galak, Jeff (2012)
- Fearing the Future of Empirical Psychology: Bem's (2011) Evidence of Psi as a Case Study of Deficiencies in Modal Research Practice — LeBel, Etienne P (2011)
- The "File Drawer Problem" and Tolerance for Null Results — Rosenthal, Robert (1979)
- Small Telescopes: Detectability and the Evaluation of Replication Results — Simonsohn, Uri (2015)
More in Skeptical
Cognitive Styles and Psi: Psi Researchers Are More Similar to Skeptics Than to Lay Believers
Searching for the Impossible: Parapsychology's Elusive Quest
False-Positive Effect in the Radin Double-Slit Experiment on Observer Consciousness as Determined with the Advanced Meta-Experimental Protocol
Cross-Examining the Case for Precognition: Comment on Mossbridge and Radin (2018)
N,N-Dimethyltryptamine and the Pineal Gland: Separating Fact from Myth
📋 Cite this paper
Francis, Gregory (2012). Too Good to Be True: Publication Bias in Two Prominent Studies from Experimental Psychology. Psychonomic Bulletin & Review. https://doi.org/10.3758/s13423-012-0227-9
@article{francis_2012_publication_bias,
title = {Too Good to Be True: Publication Bias in Two Prominent Studies from Experimental Psychology},
author = {Francis, Gregory},
year = {2012},
journal = {Psychonomic Bulletin & Review},
doi = {10.3758/s13423-012-0227-9},
}