Must Psychologists Change the Way They Analyze Their Data?
📄 Original study📌 Appears in:
Plain English Summary
When Wagenmakers and colleagues claimed a Bayesian reanalysis of Bem's precognition experiments showed no evidence for psi, Bem fired back. The crux? It all comes down to your "prior" — the assumptions baked into the math before looking at data. Wagenmakers used a broad prior placing 57% probability on huge effect sizes, which Bem's team called absurd. Swap in a realistic prior based on known effect sizes, and the result flips dramatically: from zero evidence to a Bayes factor of 13,669 — enormous support for psi. The unsettling takeaway: prior assumptions can completely reverse your conclusions.
Research Notes
Key document in the Bem FTF statistical debate. Shows how prior choice flips Bayesian conclusions from null support (Wagenmakers' Cauchy: BF = 0.63) to extreme psi support (knowledge-based: BF = 13,669). Speaks directly to Controversies #2 and #10.
A reply to Wagenmakers et al. (2011), who argued that a Bayesian reanalysis of Bem's (2011) nine precognition experiments yields no evidence for psi. Bem, Utts, and Johnson contend that Wagenmakers et al.'s diffuse Cauchy prior is unrealistic, placing 57% probability on effect sizes >= 0.8 and triggering the Lindley-Jeffreys paradox. Using a knowledge-based normal prior (90th percentile of |d| at 0.5, informed by known psychological and psi effect sizes), the combined Bayes factor across all nine experiments is 13,669 with posterior P(H0) = 7.3 x 10^-5 — extreme evidence for psi. The authors argue Bayesian methods are valuable but contain hidden traps when priors are poorly specified.
Links
Related Papers
Cites
- Feeling the Future: Experimental Evidence for Anomalous Retroactive Influences on Cognition and Affect — Bem, Daryl J (2011)
- A Bayes Factor Meta-Analysis of Bem's ESP Claim — Rouder, Jeffrey N (2011)
- Predictive Physiological Anticipation Preceding Seemingly Unpredictable Stimuli: A Meta-Analysis — Mossbridge, Julia (2012)
Cited By
- Results from a Confirmatory Replication Study of Bem (2011): Precognitive Detection of Erotic Stimuli? — Wagenmakers, Eric-Jan (2012)
- The Garden of Forking Paths: Why Multiple Comparisons Can Be a Problem, Even When There Is No "Fishing Expedition" or "P-Hacking" and the Research Hypothesis Was Posited Ahead of Time — Gelman, Andrew (2013)
Also by these authors
More in Methodology
Paranormal belief, conspiracy endorsement, and positive wellbeing: a network analysis
Planning Falsifiable Confirmatory Research
Addressing Researcher Fraud: Retrospective, Real-Time, and Preventive Strategies — Including Legal Points and Data Management That Prevents Fraud
Quantum Aspects of the Brain-Mind Relationship: A Hypothesis with Supporting Evidence
Paranormal beliefs and cognitive function: A systematic review and assessment of study quality across four decades of research
📋 Cite this paper
Bem, Daryl J, Utts, Jessica, Johnson, Wesley O (2011). Must Psychologists Change the Way They Analyze Their Data?. Journal of Personality and Social Psychology. https://doi.org/10.1037/a0024777
@article{bem_utts_johnson_2011_must_psychologists,
title = {Must Psychologists Change the Way They Analyze Their Data?},
author = {Bem, Daryl J and Utts, Jessica and Johnson, Wesley O},
year = {2011},
journal = {Journal of Personality and Social Psychology},
doi = {10.1037/a0024777},
}