Why Psychologists Must Change the Way They Analyze Their Data: The Case of Psi
β‘ Contested βπ Appears in:
Plain English Summary
When a well-known psychologist published data apparently showing people can see the future, Wagenmakers and colleagues said: not so fast. They re-crunched all nine of Daryl Bem's precognition experiments using Bayesian statistics -- a method that directly measures how much evidence supports one idea over another. The verdict was brutal: only one out of ten tests showed even modest support for psychic powers, while three actually favored the boring explanation that nothing paranormal happened. The rest were a statistical shrug. The team pinpointed where Bem went wrong: mixing up exploratory fishing expeditions with rigorous hypothesis testing, and leaning on p-values that dramatically oversell the evidence. They capped it off with a practical reform blueprint, including pre-registering studies and inviting skeptics to collaborate on experiments. This paper became a rallying cry in psychology's broader reckoning with its own statistical habits.
Abstract
Does psi exist? In a recent article, Dr. Bem conducted nine studies with over a thousand participants in an attempt to demonstrate that future events retroactively aο¬ect people's responses. Here we discuss several limitations of Bem's experiments on psi; in particular, we show that the data analysis was partly exploratory, and that one-sided p-values may overstate the statistical evidence against the null hypothesis. We reanalyze Bem's data using a default Bayesian t-test and show that the evidence for psi is weak to nonexistent. We argue that in order to convince a skeptical audience of a controversial claim, one needs to conduct strictly conο¬rmatory studies and analyze the results with statistical tests that are conservative rather than liberal. We conclude that Bem's p-values do not indicate evidence in favor of precognition; instead, they indicate that experimental psychologists need to change the way they conduct their experiments and analyze their data.
Links
Related Papers
Critiqued By
Companion
- A Bayes Factor Meta-Analysis of Bem's ESP Claim β Rouder, Jeffrey N. (2011)
- Back from the Future: Parapsychology and the Bem Affair β Alcock, James E. (2011)
- Fearing the Future of Empirical Psychology: Bem's (2011) Evidence of Psi as a Case Study of Deficiencies in Modal Research Practice β LeBel, Etienne P. (2011)
- Correcting the Past: Failures to Replicate Psi β Galak, Jeff. (2012)
- False-Positive Psychology: Undisclosed Flexibility in Data Collection and Analysis Allows Presenting Anything as Significant β Simmons, Joseph P. (2011)
- Editors' Introduction to the Special Section on Replicability in Psychological Science: A Crisis of Confidence? β Pashler, Harold. (2012)
- Measuring the Prevalence of Questionable Research Practices With Incentives for Truth Telling β John, Leslie K. (2012)
- The Garden of Forking Paths: Why Multiple Comparisons Can Be a Problem, Even When There Is No "Fishing Expedition" or "P-Hacking" and the Research Hypothesis Was Posited Ahead of Time β Gelman, Andrew. (2013)
- Failing the Future: Three Unsuccessful Attempts to Replicate Bem's 'Retroactive Facilitation of Recall' Effect β Ritchie, Stuart J. (2012)
- Results from a Confirmatory Replication Study of Bem (2011): Precognitive Detection of Erotic Stimuli? β Wagenmakers, Eric-Jan. (2012)
- Mindless Statistics β Gigerenzer, Gerd. (2004)
- Theoretical Risks and Tabular Asterisks: Sir Karl, Sir Ronald, and the Slow Progress of Soft Psychology β Meehl, Paul E. (1978)
- Bayesian t tests for accepting and rejecting the null hypothesis β Rouder, Jeffrey N. (2009)
- Equivalence Tests: A Practical Primer for t Tests, Correlations, and Meta-Analyses β Lakens, DaniΓ«l. (2017)
- We Should Have Seen This Coming β Schwarzkopf, D. Samuel. (2014)
Extends
Cites
- Meta-Analysis of Free-Response Studies, 1992β2008: Assessing the Noise Reduction Model in Parapsychology β Storm, Lance. (2010)
- Replication and Meta-Analysis in Parapsychology β Utts, Jessica. (1991)
- Why Is Psi So Elusive? A Review and Proposed Model β Kennedy, James E. (2001)
- Why Most Published Research Findings Are False β Ioannidis, John P.A. (2005)
- Experimenter Effects and the Remote Detection of Staring β Wiseman, Richard. (1997)
Cited By
- Cross-Examining the Case for Precognition: Comment on Mossbridge and Radin (2018) β Houran, James. (2018)
- Bem's 'Feeling the Future' (2011) Five Years Later: Its Impact on Scientific Literature β Silva, Bruno A. (2017)
- Why Most Research Findings About Psi Are False: The Replicability Crisis, the Psi Paradox and the Myth of Sisyphus β Rabeyron, Thomas. (2020)
- Equivalence Tests: A Practical Primer for t Tests, Correlations, and Meta-Analyses β Lakens, DaniΓ«l. (2017)
- The Experimental Evidence for Parapsychological Phenomena: A Review β CardeΓ±a, Etzel. (2018)
- The Garden of Forking Paths: Why Multiple Comparisons Can Be a Problem, Even When There Is No "Fishing Expedition" or "P-Hacking" and the Research Hypothesis Was Posited Ahead of Time β Gelman, Andrew. (2013)
- Bayesian and Classical Hypothesis Testing: Practical Differences for a Controversial Area of Research β Kennedy, J. E. (2014)
- Replication Unreliability in Psychology: Elusive Phenomena or "Elusive" Statistical Power? β Tressoldi, Patrizio E. (2012)
- Scientific Utopia: II. Restructuring Incentives and Practices to Promote Truth Over Publishability β Nosek, Brian A. (2012)
- Feeling the Future: A Meta-Analysis of 90 Experiments on the Anomalous Anticipation of Random Future Events β Bem, Daryl J. (2015)
- Too Good to Be True: Publication Bias in Two Prominent Studies from Experimental Psychology β Francis, Gregory. (2012)
- Correcting the Past: Failures to Replicate Psi β Galak, Jeff. (2012)
- Fearing the Future of Empirical Psychology: Bem's (2011) Evidence of Psi as a Case Study of Deficiencies in Modal Research Practice β LeBel, Etienne P. (2011)
- Feeling the Future Again: Retroactive Avoidance of Negative Stimuli β Maier, Markus A. (2014)
- A Preregistered Multi-Lab Replication of Maier et al. (2014, Exp. 4) Testing Retroactive Avoidance β Maier, Markus A. (2020)
- Explicit Anomalous Cognition: A Review of the Best Evidence in Ganzfeld, Forced-choice, Remote Viewing and Dream Studies β Baptista, Johann. (2015)
- Precognition as a Form of Prospection: A Review of the Evidence β Mossbridge, Julia A. (2018)
- Cognitive Styles and Psi: Psi Researchers Are More Similar to Skeptics Than to Lay Believers β Pehlivanova, M. (2024)
- Failing the Future: Three Unsuccessful Attempts to Replicate Bem's 'Retroactive Facilitation of Recall' Effect β Ritchie, Stuart J. (2012)
- A Bayes Factor Meta-Analysis of Recent Extrasensory Perception Experiments: Comment on Storm, Tressoldi, and Di Risio (2010) β Rouder, Jeffrey N. (2013)
- A Bayes Factor Meta-Analysis of Bem's ESP Claim β Rouder, Jeffrey N. (2011)
- Retro-priming, priming, and double testing: psi and replication in a testβretest design β Rabeyron, Thomas. (2014)
- We Should Have Seen This Coming β Schwarzkopf, D. Samuel. (2014)
- Evidence for Anomalistic Correlations Between Human Behavior and a Random Event Generator: Result of an Independent Replication of a Micro-PK Experiment β Walach, Harald. (2020)
- Brain-to-Brain (Mind-to-Mind) Interaction at Distance: A Confirmatory Study β Tressoldi, Patrizio E. (2014)
- Future directions in precognition research: more research can bridge the gap between skeptics and proponents β Franklin, Michael S. (2014)
- Raising the value of research studies in psychological science by increasing the credibility of research reports: the transparent Psi project β Kekecs, Zoltan. (2023)
Same Research Program
More in Skeptical
Searching for the Impossible: Parapsychology's Elusive Quest
False-Positive Effect in the Radin Double-Slit Experiment on Observer Consciousness as Determined with the Advanced Meta-Experimental Protocol
N,N-Dimethyltryptamine and the Pineal Gland: Separating Fact from Myth
Paranormal psychic believers and skeptics: a large-scale test of the cognitive differences hypothesis
Meta-Analyses Are No Substitute for Registered Replications: A Skeptical Perspective on Religious Priming
π Cite this paper
Wagenmakers, Eric-Jan., Wetzels, Ruud., Borsboom, Denny., van der Maas, Han. (2011). Why Psychologists Must Change the Way They Analyze Their Data: The Case of Psi. Journal of Personality and Social Psychology. https://doi.org/10.1037/a0022790
@article{wagenmakers_2011_why_psychologists,
title = {Why Psychologists Must Change the Way They Analyze Their Data: The Case of Psi},
author = {Wagenmakers, Eric-Jan. and Wetzels, Ruud. and Borsboom, Denny. and van der Maas, Han.},
year = {2011},
journal = {Journal of Personality and Social Psychology},
doi = {10.1037/a0022790},
}