The "File Drawer Problem" and Tolerance for Null Results
📄 Original study ↗Plain English Summary
Imagine scientists only publishing their exciting findings while stuffing boring "nothing happened" results into a file drawer. That's the "file drawer problem," and this landmark paper tackled it head-on. Rosenthal invented a clever tool called the "fail-safe N" — a formula that calculates how many hidden, unpublished null studies would need to exist in those file drawers to wipe out a published positive finding. The numbers can be staggering: in one example, you'd need over 3,200 buried studies to overturn 94 published ones, and nearly 50,000 to cancel out 311. He proposed a handy rule of thumb for when results are sturdy enough to trust despite possible hidden studies. This formula became absolutely essential in parapsychology research, where every meta-analysis now uses it to argue whether cumulative evidence for psychic phenomena can survive the file drawer threat.
Research Notes
One of the most cited methodology papers in behavioral science. The fail-safe N became standard practice in meta-analysis and is reported in virtually every psi meta-analysis in this library (Radin 1989, Bem 1994, Storm 2010, etc.). Its limitations are central to the meta-debate about whether cumulative psi evidence is robust to publication bias.
Introduces the "file drawer problem" — the concern that journals publish the 5% of studies showing Type I errors while 95% of null results remain unpublished — and derives a quantitative solution: the fail-safe N (tolerance for future null results). Using the method of adding standard normal deviates across k independent studies, the formula X = (k/2.706)[k(Z-bar)^2 - 2.706] computes how many additional null-result studies would be needed to reduce a combined significance level to p = .05. Illustrated with interpersonal expectancy research: 94 studies require 3,263 null studies to overturn; 311 studies require 49,457. Proposes X >= 5k + 10 as a threshold for file-drawer resistance.
Links
Related Papers
Companion
- Why Most Published Research Findings Are False — Ioannidis, John P.A (2005)
- Too Good to Be True: Publication Bias in Two Prominent Studies from Experimental Psychology — Francis, Gregory (2012)
- Estimating the Reproducibility of Psychological Science — Open Science Collaboration (2015)
- Power failure: why small sample size undermines the reliability of neuroscience — Button, Katherine S (2013)
- Scientists behaving badly — Martinson, Brian C (2005)
Cited By
- Evidence for Consciousness-Related Anomalies in Random Physical Systems — Radin, Dean I (1989)
- Does Psi Exist? Replicable Evidence for an Anomalous Process of Information Transfer — Bem, Daryl J (1994)
- "Future Telling": A Meta-Analysis of Forced-Choice Precognition Experiments, 1935-1987 — Honorton, Charles (1989)
- Examining Psychokinesis: The Interaction of Human Intention With Random Number Generators—A Meta-Analysis — Bösch, Holger (2006)
- Why Science Is Not Necessarily Self-Correcting — Ioannidis, John P.A (2012)
- Testing for Questionable Research Practices in a Meta-Analysis: An Example from Experimental Parapsychology — Bierman, Dick J (2016)
- Scientific Utopia: II. Restructuring Incentives and Practices to Promote Truth Over Publishability — Nosek, Brian A (2012)
More in Methodology
Paranormal belief, conspiracy endorsement, and positive wellbeing: a network analysis
Planning Falsifiable Confirmatory Research
Addressing Researcher Fraud: Retrospective, Real-Time, and Preventive Strategies — Including Legal Points and Data Management That Prevents Fraud
Quantum Aspects of the Brain-Mind Relationship: A Hypothesis with Supporting Evidence
Paranormal beliefs and cognitive function: A systematic review and assessment of study quality across four decades of research
📋 Cite this paper
Rosenthal, Robert (1979). The "File Drawer Problem" and Tolerance for Null Results. Psychological Bulletin. https://doi.org/10.1037/0033-2909.86.3.638
@article{rosenthal_1979_file_drawer,
title = {The "File Drawer Problem" and Tolerance for Null Results},
author = {Rosenthal, Robert},
year = {1979},
journal = {Psychological Bulletin},
doi = {10.1037/0033-2909.86.3.638},
}