Testing for Questionable Research Practices in a Meta-Analysis: An Example from Experimental Parapsychology
📄 Original study ↗📌 Appears in:
Plain English Summary
Scientists sometimes cut corners -- peeking at data early, tweaking analyses until something looks good. These 'questionable research practices' (QRPs) can inflate results across a field. So how much do they explain Ganzfeld telepathy experiments, where subjects score 31% when chance is 25%? This study simulated seven QRPs at realistic rates. The headline: QRPs account for roughly 60% of the effect -- hefty! But even after scrubbing that noise away, a small but stubbornly significant residual remains. This makes the paper a rare honest broker, giving ammunition to both skeptics and proponents: the evidence is messy but not fully explainable by sloppy methods.
Research Notes
First systematic simulation of multiple QRPs' combined impact on a parapsychological meta-analysis. Central to the ganzfeld telepathy (#1) and meta-debate (#10) controversies. The conclusion that QRPs explain ~60% but not all of the effect makes this relevant to both pro-psi and skeptical positions.
Using Monte Carlo simulations and a genetic algorithm, a method was developed to quantify the impact of Questionable Research Practices (QRPs) on meta-analytic results. Applied to 78 post-1985 Ganzfeld telepathy experiments (3,494 trials, mean hit rate 31% vs. 25% chance), seven QRPs were modeled at prevalence rates from published surveys of psychologists. With realistic QRP parameters and no anomalous effect, simulations failed to reproduce the empirical database (F=10.15, p<0.05). Allowing a 2% excess hit rate yielded acceptable fit (F=1.79, p=0.47). QRPs explain approximately 60% of the reported effect size, but a residual effect remains significant (p=0.003).
Links
Related Papers
Cites
- Meta-Analysis of Free-Response Studies, 1992–2008: Assessing the Noise Reduction Model in Parapsychology — Storm, Lance (2010)
- Meta-Analysis That Conceals More Than It Reveals: Comment on Storm et al. (2010) — Hyman, Ray (2010)
- False-Positive Psychology: Undisclosed Flexibility in Data Collection and Analysis Allows Presenting Anything as Significant — Simmons, Joseph P (2011)
- Why Most Published Research Findings Are False — Ioannidis, John P.A (2005)
- Estimating the Reproducibility of Psychological Science — Open Science Collaboration (2015)
- Examining Psychokinesis: The Interaction of Human Intention With Random Number Generators—A Meta-Analysis — Bösch, Holger (2006)
- A Joint Communiqué: The Psi Ganzfeld Controversy — Hyman, Ray (1986)
- The "File Drawer Problem" and Tolerance for Null Results — Rosenthal, Robert (1979)
Companion
- Power failure: why small sample size undermines the reliability of neuroscience — Button, Katherine S (2013)
- Can Parapsychology Move Beyond the Controversies of Retrospective Meta-Analyses? — Kennedy, J.E (2013)
- Experimenter Fraud: What Are Appropriate Methodological Standards? — Kennedy, J.E (2017)
- Is the Methodological Revolution in Psychology Over or Just Beginning? — Kennedy, J.E (2016)
- Religious Priming: A Meta-Analysis With a Focus on Prosociality — Shariff, Azim F (2015)
Also by these authors
More in Methodology
Paranormal belief, conspiracy endorsement, and positive wellbeing: a network analysis
Planning Falsifiable Confirmatory Research
Addressing Researcher Fraud: Retrospective, Real-Time, and Preventive Strategies — Including Legal Points and Data Management That Prevents Fraud
Quantum Aspects of the Brain-Mind Relationship: A Hypothesis with Supporting Evidence
Paranormal beliefs and cognitive function: A systematic review and assessment of study quality across four decades of research
📋 Cite this paper
Bierman, Dick J, Spottiswoode, James P, Bijl, Aron (2016). Testing for Questionable Research Practices in a Meta-Analysis: An Example from Experimental Parapsychology. PLOS ONE. https://doi.org/10.1371/journal.pone.0153049
@article{bierman_2016_questionable_practices,
title = {Testing for Questionable Research Practices in a Meta-Analysis: An Example from Experimental Parapsychology},
author = {Bierman, Dick J and Spottiswoode, James P and Bijl, Aron},
year = {2016},
journal = {PLOS ONE},
doi = {10.1371/journal.pone.0153049},
}