Must Psychologists Change the Way They Analyze Their Data?
⚡ Contested📌 Appears in:
Plain English Summary
When Wagenmakers and colleagues claimed a Bayesian reanalysis of Bem's precognition experiments showed no evidence for psi, Bem fired back. The crux? It all comes down to your "prior" — the assumptions baked into the math before looking at data. Wagenmakers used a broad prior placing 57% probability on huge effect sizes, which Bem's team called absurd. Swap in a realistic prior based on known effect sizes, and the result flips dramatically: from zero evidence to a Bayes factor of 13,669 — enormous support for psi. The unsettling takeaway: prior assumptions can completely reverse your conclusions.
Abstract
Wagenmakers, Wetzels, Borsboom, and van der Maas (2011) argued that psychologists should replace the familiar "frequentist" statistical analyses of their data with Bayesian analyses. To illustrate their argument, they reanalyzed a set of psi experiments published recently in this journal by Bem (2011), maintaining that, contrary to his conclusion, his data do not yield evidence in favor of the psi hypothesis. We argue that they have incorrectly selected an unrealistic prior distribution for their analysis and that a Bayesian analysis using a more reasonable distribution yields strong evidence in favor of the psi hypothesis. More generally, we argue that there are advantages to Bayesian analyses that merit their increased use in the future. However, as Wagenmakers et al.'s analysis inadvertently revealed, they contain hidden traps that must be better understood before being more widely substituted for the familiar frequentist analyses currently employed by most research psychologists.
Links
Related Papers
Cites
Cited By
- Results from a Confirmatory Replication Study of Bem (2011): Precognitive Detection of Erotic Stimuli? — Wagenmakers, Eric-Jan. (2012)
- The Garden of Forking Paths: Why Multiple Comparisons Can Be a Problem, Even When There Is No "Fishing Expedition" or "P-Hacking" and the Research Hypothesis Was Posited Ahead of Time — Gelman, Andrew. (2013)
- Too Good to Be True: Publication Bias in Two Prominent Studies from Experimental Psychology — Francis, Gregory. (2012)
- Feeling the Future Again: Retroactive Avoidance of Negative Stimuli — Maier, Markus A. (2014)
- A Preregistered Multi-Lab Replication of Maier et al. (2014, Exp. 4) Testing Retroactive Avoidance — Maier, Markus A. (2020)
- Precognition as a Form of Prospection: A Review of the Evidence — Mossbridge, Julia A. (2018)
- We Should Have Seen This Coming — Schwarzkopf, D. Samuel. (2014)
- An Agenda for Purely Confirmatory Research — Wagenmakers, Eric-Jan. (2012)
- Raising the value of research studies in psychological science by increasing the credibility of research reports: the transparent Psi project — Kekecs, Zoltan. (2023)
Companion
- An Agenda for Purely Confirmatory Research — Wagenmakers, Eric-Jan. (2012)
- Bem's 'Feeling the Future' (2011) Five Years Later: Its Impact on Scientific Literature — Silva, Bruno A. (2017)
- Predictive Physiological Anticipation Preceding Seemingly Unpredictable Stimuli: A Meta-Analysis — Mossbridge, Julia. (2012)
- Feeling the Future: Experimental Evidence for Anomalous Retroactive Influences on Cognition and Affect — Bem, Daryl J. (2011)
Also by these authors
More in Methodology
Paranormal belief, conspiracy endorsement, and positive wellbeing: a network analysis
Planning Falsifiable Confirmatory Research
Addressing Researcher Fraud: Retrospective, Real-Time, and Preventive Strategies — Including Legal Points and Data Management That Prevents Fraud
Quantum Aspects of the Brain-Mind Relationship: A Hypothesis with Supporting Evidence
Paranormal beliefs and cognitive function: A systematic review and assessment of study quality across four decades of research
📋 Cite this paper
Bem, Daryl J., Utts, Jessica., Johnson, Wesley O. (2011). Must Psychologists Change the Way They Analyze Their Data?. Journal of Personality and Social Psychology. https://doi.org/10.1037/a0024777
@article{bem_utts_johnson_2011_must_psychologists,
title = {Must Psychologists Change the Way They Analyze Their Data?},
author = {Bem, Daryl J. and Utts, Jessica. and Johnson, Wesley O.},
year = {2011},
journal = {Journal of Personality and Social Psychology},
doi = {10.1037/a0024777},
}