Meta-Analyses Are No Substitute for Registered Replications: A Skeptical Perspective on Religious Priming
⚡ Contested ↗📌 Appears in:
Plain English Summary
A fun detective story about statistics fighting each other. A previous team combined 92 studies and concluded that reminding people about religion ('religious priming') makes them more generous. This team of heavyweight statisticians re-crunched the same data using two methods for detecting publication bias (the tendency for only exciting results to get published). One method found zero effect. The other found a solid one. Same data, opposite answers! The takeaway: no clever math on old studies can settle a controversial question. The only fix is brand-new, pre-planned experiments where researchers commit to their methods upfront.
Research Notes
Central methodological paper in meta-debate controversy. Authors include prominent Bayesian statisticians (Wagenmakers, Vandekerckhove) arguing against reliance on meta-analysis for contested effects. Directly supports position that psi research needs registered replications rather than retrospective meta-analyses. Cited by subsequent methodology papers discussing publication bias and replication crisis.
Critique of Shariff et al. (2015) meta-analysis claiming religious priming has small but reliable effect on prosocial behavior. Re-analyzes the same 92-study dataset using PET-PEESE and Bayesian bias correction methods. PET-PEESE finds no evidence for effect after correcting for publication bias (intercept = -0.002, p = 0.97); BBC method reaches opposite conclusion with strong evidence for real effect (~0.3). Argues contradictory results demonstrate meta-analysis alone cannot resolve disputed effects due to inability to disentangle true effects from publication bias and experimenter bias. Concludes preregistered large-scale replications are the sole remedy.
Links
Related Papers
Also by these authors
Editors' Introduction to the Special Section on Replicability in Psychological Science: A Crisis of Confidence?
An Agenda for Purely Confirmatory Research
Results from a Confirmatory Replication Study of Bem (2011): Precognitive Detection of Erotic Stimuli?
More in Skeptical
Cognitive Styles and Psi: Psi Researchers Are More Similar to Skeptics Than to Lay Believers
Searching for the Impossible: Parapsychology's Elusive Quest
False-Positive Effect in the Radin Double-Slit Experiment on Observer Consciousness as Determined with the Advanced Meta-Experimental Protocol
Cross-Examining the Case for Precognition: Comment on Mossbridge and Radin (2018)
N,N-Dimethyltryptamine and the Pineal Gland: Separating Fact from Myth
📋 Cite this paper
van Elk, Michiel, Matzke, Dora, Gronau, Quentin F, Guan, Maime, Vandekerckhove, Joachim, Wagenmakers, Eric-Jan (2015). Meta-Analyses Are No Substitute for Registered Replications: A Skeptical Perspective on Religious Priming. Frontiers in Psychology. https://doi.org/10.3389/fpsyg.2015.01365
@article{van_elk_2015_registered_replications,
title = {Meta-Analyses Are No Substitute for Registered Replications: A Skeptical Perspective on Religious Priming},
author = {van Elk, Michiel and Matzke, Dora and Gronau, Quentin F and Guan, Maime and Vandekerckhove, Joachim and Wagenmakers, Eric-Jan},
year = {2015},
journal = {Frontiers in Psychology},
doi = {10.3389/fpsyg.2015.01365},
}