Plain English Summary
This paper lays out a devastating case against the p-value, the workhorse statistic that scientists use to decide whether a result is 'real.' Wagenmakers shows three damning problems: p-values depend on imaginary data you never collected, they change based on what the researcher intended to do (the same data can be 'significant' or not depending on when you planned to stop collecting), and identical p-values can mean wildly different things at different sample sizes. Here's the kicker: at a commonly used threshold, the probability that the boring null hypothesis is actually true can range from 69% to a whopping 92% as your sample grows. As a fix, Wagenmakers champions the Bayesian information criterion (BIC), a straightforward alternative that approximates Bayesian reasoning without the heavy mathematical machinery. This paper later became a loaded weapon in debates over claimed evidence for psychic phenomena.
Research Notes
Foundational paper for the Bayesian statistics movement in psychology. Wagenmakers later applied these exact BIC and Bayes factor arguments to critique Bem's (2011) 'Feeling the Future' precognition experiments, making this a direct precursor to the most prominent statistical controversy in this library.
Three fundamental problems with p value null-hypothesis significance testing (NHST) are reviewed with concrete examples. First, p values depend on data never observed, violating the conditionality principle. Second, p values depend on the researcher's subjective sampling intentions — identical data yield p = 0.146 under binomial but p = 0.033 under negative binomial sampling. Third, the 'p postulate' (equal p values = equal evidence) is false: Bayesian analysis shows that for data with p = .05, posterior probability of H₀ rises from ~0.69 at n = 400 to ~0.92 at n = 10,000. The Bayesian information criterion (BIC) is proposed as a practical alternative, approximating Bayesian hypothesis testing without requiring prior specification.
Links
Related Papers
Extended By
Companion
- A Bayes Factor Meta-Analysis of Bem's ESP Claim — Rouder, Jeffrey N (2011)
- Power failure: why small sample size undermines the reliability of neuroscience — Button, Katherine S (2013)
- Mindless Statistics — Gigerenzer, Gerd (2004)
- Bayesian and Classical Hypothesis Testing: Practical Differences for a Controversial Area of Research — Kennedy, J.E (2014)
- Equivalence Tests: A Practical Primer for t Tests, Correlations, and Meta-Analyses — Lakens, Daniël (2017)
Also by these authors
More in Methodology
Paranormal belief, conspiracy endorsement, and positive wellbeing: a network analysis
Planning Falsifiable Confirmatory Research
Addressing Researcher Fraud: Retrospective, Real-Time, and Preventive Strategies — Including Legal Points and Data Management That Prevents Fraud
Quantum Aspects of the Brain-Mind Relationship: A Hypothesis with Supporting Evidence
Paranormal beliefs and cognitive function: A systematic review and assessment of study quality across four decades of research
📋 Cite this paper
Wagenmakers, Eric-Jan (2007). A Practical Solution to the Pervasive Problems of p Values. Psychonomic Bulletin & Review. https://doi.org/10.3758/BF03194105
@article{wagenmakers_2007_practical_solution,
title = {A Practical Solution to the Pervasive Problems of p Values},
author = {Wagenmakers, Eric-Jan},
year = {2007},
journal = {Psychonomic Bulletin & Review},
doi = {10.3758/BF03194105},
}