False-Positive Psychology: Undisclosed Flexibility in Data Collection and Analysis Allows Presenting Anything as Significant
⚡ Contested ↗📌 Appears in:
Plain English Summary
One of the most important — and funniest — papers in modern science. Published alongside a famous study claiming to prove psychic powers, it asked: how easy is it to get a significant result when nothing real is happening? Terrifyingly easy. Simulating 15,000 samples, the authors showed common research shortcuts — choosing which measurements to report, peeking at data mid-collection, picking control variables — inflate false-positive rates from 5% to between 7.7% and 12.6% individually. Combine all four and you hit a jaw-dropping 61%. To prove it, they produced "significant" evidence a Beatles song makes people literally younger. The paper helped launch the pre-registration movement and became a blueprint for transparent research.
Abstract
In this article, we accomplish two things. First, we show that despite empirical psychologists' nominal endorsement of a low rate of false-positive findings (≤ .05), flexibility in data collection, analysis, and reporting dramatically increases actual false-positive rates. In many cases, a researcher is more likely to falsely find evidence that an effect exists than to correctly find evidence that it does not. We present computer simulations and a pair of actual experiments that demonstrate how unacceptably easy it is to accumulate (and report) statistically significant evidence for a false hypothesis. Second, we suggest a simple, low-cost, and straightforwardly effective disclosure-based solution to this problem. The solution involves six concrete requirements for authors and four guidelines for reviewers, all of which impose a minimal burden on the publication process.
Links
Related Papers
Companion
- The Garden of Forking Paths: Why Multiple Comparisons Can Be a Problem, Even When There Is No "Fishing Expedition" or "P-Hacking" and the Research Hypothesis Was Posited Ahead of Time — Gelman, Andrew. (2013)
- Why Most Published Research Findings Are False — Ioannidis, John P.A. (2005)
- Why Psychologists Must Change the Way They Analyze Their Data: The Case of Psi — Wagenmakers, Eric-Jan. (2011)
- Bayesian and Classical Hypothesis Testing: Practical Differences for a Controversial Area of Research — Kennedy, J. E. (2014)
- Commentary: Reproducibility in Psychological Science: When Do Psychological Phenomena Exist? — Heino, Matti T. J. (2017)
- Estimating the Reproducibility of Psychological Science — Open Science Collaboration (2015)
- Fearing the Future of Empirical Psychology: Bem's (2011) Evidence of Psi as a Case Study of Deficiencies in Modal Research Practice — LeBel, Etienne P. (2011)
- Failing the Future: Three Unsuccessful Attempts to Replicate Bem's 'Retroactive Facilitation of Recall' Effect — Ritchie, Stuart J. (2012)
- Back from the Future: Parapsychology and the Bem Affair — Alcock, James E. (2011)
- Investigating variation in replicability: A "many labs" replication project — Klein, Richard A. (2014)
Cited By
- Editors' Introduction to the Special Section on Replicability in Psychological Science: A Crisis of Confidence? — Pashler, Harold. (2012)
- Too Good to Be True: Publication Bias in Two Prominent Studies from Experimental Psychology — Francis, Gregory. (2012)
- Estimating the Reproducibility of Psychological Science — Open Science Collaboration (2015)
- An Agenda for Purely Confirmatory Research — Wagenmakers, Eric-Jan. (2012)
- Measuring the Prevalence of Questionable Research Practices With Incentives for Truth Telling — John, Leslie K. (2012)
- Addressing Researcher Fraud: Retrospective, Real-Time, and Preventive Strategies — Including Legal Points and Data Management That Prevents Fraud — Kennedy, James E. (2024)
- Power failure: why small sample size undermines the reliability of neuroscience — Button, Katherine S. (2013)
- Testing for Questionable Research Practices in a Meta-Analysis: An Example from Experimental Parapsychology — Bierman, Dick J. (2016)
- Can Parapsychology Move Beyond the Controversies of Retrospective Meta-Analyses? — Kennedy, J.E. (2013)
- The Garden of Forking Paths: Why Multiple Comparisons Can Be a Problem, Even When There Is No "Fishing Expedition" or "P-Hacking" and the Research Hypothesis Was Posited Ahead of Time — Gelman, Andrew. (2013)
- Decline Effects: Types, Mechanisms, and Personal Reflections — Protzko, John. (2017)
- Religious Priming: A Meta-Analysis With a Focus on Prosociality — Shariff, Azim F. (2015)
- Why Science Is Not Necessarily Self-Correcting — Ioannidis, John P. A. (2012)
- Scientific Utopia: II. Restructuring Incentives and Practices to Promote Truth Over Publishability — Nosek, Brian A. (2012)
- Investigating variation in replicability: A "many labs" replication project — Klein, Richard A. (2014)
- Correcting the Past: Failures to Replicate Psi — Galak, Jeff. (2012)
- Planning Falsifiable Confirmatory Research — Kennedy, James E. (2024)
- Feeling the Future Again: Retroactive Avoidance of Negative Stimuli — Maier, Markus A. (2014)
- A Preregistered Multi-Lab Replication of Maier et al. (2014, Exp. 4) Testing Retroactive Avoidance — Maier, Markus A. (2020)
More in Skeptical
Cognitive Styles and Psi: Psi Researchers Are More Similar to Skeptics Than to Lay Believers
Searching for the Impossible: Parapsychology's Elusive Quest
False-Positive Effect in the Radin Double-Slit Experiment on Observer Consciousness as Determined with the Advanced Meta-Experimental Protocol
Cross-Examining the Case for Precognition: Comment on Mossbridge and Radin (2018)
N,N-Dimethyltryptamine and the Pineal Gland: Separating Fact from Myth
📋 Cite this paper
Simmons, Joseph P., Nelson, Leif D., Simonsohn, Uri (2011). False-Positive Psychology: Undisclosed Flexibility in Data Collection and Analysis Allows Presenting Anything as Significant. Psychological Science. https://doi.org/10.1177/0956797611417632
@article{simmons_2011_false_positive,
title = {False-Positive Psychology: Undisclosed Flexibility in Data Collection and Analysis Allows Presenting Anything as Significant},
author = {Simmons, Joseph P. and Nelson, Leif D. and Simonsohn, Uri},
year = {2011},
journal = {Psychological Science},
doi = {10.1177/0956797611417632},
}