False-Positive Psychology: Undisclosed Flexibility in Data Collection and Analysis Allows Presenting Anything as Significant
⚡ Contested ↗📌 Appears in:
Plain English Summary
One of the most important — and funniest — papers in modern science. Published alongside a famous study claiming to prove psychic powers, it asked: how easy is it to get a significant result when nothing real is happening? Terrifyingly easy. Simulating 15,000 samples, the authors showed common research shortcuts — choosing which measurements to report, peeking at data mid-collection, picking control variables — inflate false-positive rates from 5% to nearly 13% each. Combine all four and you hit a jaw-dropping 61%. To prove it, they produced "significant" evidence a Beatles song makes people literally younger. The paper helped launch the pre-registration movement and became a blueprint for transparent research.
Research Notes
Foundational paper for the replication crisis, published the same year as Bem's (2011) precognition study. Coined 'researcher degrees of freedom' and demonstrated through simulation and absurdist experiment how easy it is to find significance when none exists. The proposed disclosure requirements were precursors to the pre-registration movement. Directly relevant to evaluating all psi studies involving analytic flexibility.
Demonstrates that flexibility in data collection, analysis, and reporting dramatically inflates false-positive rates beyond the nominal 5%. Monte Carlo simulations of 15,000 samples show four common researcher degrees of freedom -- flexible dependent variables, optional stopping, covariate selection, and condition dropping -- individually raise false-positive rates to 7.7-12.6% and in combination produce a 60.7% rate. Two actual experiments exploit these freedoms to produce statistically significant evidence for an impossible hypothesis (that listening to a Beatles song makes people younger, p=.040). Proposes six author disclosure requirements and four reviewer guidelines as remedy.
Links
Related Papers
Companion
- The Garden of Forking Paths: Why Multiple Comparisons Can Be a Problem, Even When There Is No "Fishing Expedition" or "P-Hacking" and the Research Hypothesis Was Posited Ahead of Time — Gelman, Andrew (2013)
- Why Most Published Research Findings Are False — Ioannidis, John P.A (2005)
- Why Psychologists Must Change the Way They Analyze Their Data: The Case of Psi — Wagenmakers, Eric-Jan (2011)
- Bayesian and Classical Hypothesis Testing: Practical Differences for a Controversial Area of Research — Kennedy, J.E (2014)
- Commentary: Reproducibility in Psychological Science: When Do Psychological Phenomena Exist? — Heino, Matti T. J (2017)
- Estimating the Reproducibility of Psychological Science — Open Science Collaboration (2015)
- Fearing the Future of Empirical Psychology: Bem's (2011) Evidence of Psi as a Case Study of Deficiencies in Modal Research Practice — LeBel, Etienne P (2011)
- Failing the Future: Three Unsuccessful Attempts to Replicate Bem's 'Retroactive Facilitation of Recall' Effect — Ritchie, Stuart J (2012)
Cited By
- Editors' Introduction to the Special Section on Replicability in Psychological Science: A Crisis of Confidence? — Pashler, Harold (2012)
- Too Good to Be True: Publication Bias in Two Prominent Studies from Experimental Psychology — Francis, Gregory (2012)
- Estimating the Reproducibility of Psychological Science — Open Science Collaboration (2015)
- An Agenda for Purely Confirmatory Research — Wagenmakers, Eric-Jan (2012)
- Measuring the Prevalence of Questionable Research Practices With Incentives for Truth Telling — John, Leslie K (2012)
- Addressing Researcher Fraud: Retrospective, Real-Time, and Preventive Strategies — Including Legal Points and Data Management That Prevents Fraud — Kennedy, James E (2024)
- Power failure: why small sample size undermines the reliability of neuroscience — Button, Katherine S (2013)
- Testing for Questionable Research Practices in a Meta-Analysis: An Example from Experimental Parapsychology — Bierman, Dick J (2016)
- Can Parapsychology Move Beyond the Controversies of Retrospective Meta-Analyses? — Kennedy, J.E (2013)
- The Garden of Forking Paths: Why Multiple Comparisons Can Be a Problem, Even When There Is No "Fishing Expedition" or "P-Hacking" and the Research Hypothesis Was Posited Ahead of Time — Gelman, Andrew (2013)
- Decline Effects: Types, Mechanisms, and Personal Reflections — Protzko, John (2017)
- Religious Priming: A Meta-Analysis With a Focus on Prosociality — Shariff, Azim F (2015)
- Why Science Is Not Necessarily Self-Correcting — Ioannidis, John P.A (2012)
- Scientific Utopia: II. Restructuring Incentives and Practices to Promote Truth Over Publishability — Nosek, Brian A (2012)
Also by these authors
More in Skeptical
Cognitive Styles and Psi: Psi Researchers Are More Similar to Skeptics Than to Lay Believers
Searching for the Impossible: Parapsychology's Elusive Quest
False-Positive Effect in the Radin Double-Slit Experiment on Observer Consciousness as Determined with the Advanced Meta-Experimental Protocol
Cross-Examining the Case for Precognition: Comment on Mossbridge and Radin (2018)
N,N-Dimethyltryptamine and the Pineal Gland: Separating Fact from Myth
📋 Cite this paper
Simmons, Joseph P, Nelson, Leif D, Simonsohn, Uri (2011). False-Positive Psychology: Undisclosed Flexibility in Data Collection and Analysis Allows Presenting Anything as Significant. Psychological Science. https://doi.org/10.1177/0956797611417632
@article{simmons_2011_false_positive,
title = {False-Positive Psychology: Undisclosed Flexibility in Data Collection and Analysis Allows Presenting Anything as Significant},
author = {Simmons, Joseph P and Nelson, Leif D and Simonsohn, Uri},
year = {2011},
journal = {Psychological Science},
doi = {10.1177/0956797611417632},
}