Meta-Analysis That Conceals More Than It Reveals: Comment on Storm et al. (2010)
🧐 Skeptical/Critical ↗📌 Appears in:
Plain English Summary
Meta-analysis is a technique for pooling results from many studies to find the big picture -- and Ray Hyman thinks it can be a magician's trick when applied to psychic research. Responding to a study claiming strong evidence for telepathy in ganzfeld experiments (where a receiver in a relaxed, sensory-reduced state tries to pick up mental images from a sender), Hyman dug into the numbers and found something revealing. Nearly all the impressive hit rates came from just four researchers, while everyone else scored at chance. When the experiments used video clips as targets, results looked great, but with still images -- the kind used in the original studies -- performance was flat. Most damning: the newest, most rigorous round of experiments hit right at the 25% rate you'd expect from pure guessing. Hyman's takeaway: don't let averaging across messy, inconsistent data create an illusion of proof.
Research Notes
Key skeptical commentary in the Psychological Bulletin ganzfeld exchange. Introduces the N-rays analogy — that meta-analysis of a nonexistent phenomenon can still yield significant composite effect sizes — as a central challenge to meta-analytic defenses of psi. Part of Hyman's decades-long critical engagement with ganzfeld evidence.
Responding to Storm, Tressoldi, and Di Risio's (2010) meta-analysis of ganzfeld studies, this commentary argues that meta-analytic aggregation manufactures apparent consistency from fundamentally heterogeneous data. The original ganzfeld database's significant hit rate derived almost entirely from four experimenters (44% hit rate) while others obtained chance-level results (26%). The autoganzfeld's significance came only from dynamic targets (37%), with static targets at chance (~26%), constituting a failed replication of the original static-target database. Autoganzfeld II, meeting all of Storm et al.'s criteria for a reliable study, yielded hit rates of 26.5% (N=151) and 25.8% (N=209) — chance level. Hyman concludes that parapsychology requires prospective, independently replicable evidence rather than retrospective meta-analytic consistency.
Links
Related Papers
Same Research Program
- Anomaly or Artifact? Comments on Bem and Honorton — Hyman, Ray (1994)
- A Joint Communiqué: The Psi Ganzfeld Controversy — Hyman, Ray (1986)
- Evaluation of a Program on Anomalous Mental Phenomena — Hyman, Ray (1996)
- Parapsychological Research: A Tutorial Review and Critical Appraisal — Hyman, Ray (1986)
Cites
- Does Psi Exist? Replicable Evidence for an Anomalous Process of Information Transfer — Bem, Daryl J (1994)
- Why Is Psi So Elusive? A Review and Proposed Model — Kennedy, James E (2001)
- The Capricious, Actively Evasive, Unsustainable Nature of Psi: A Summary and Hypotheses — Kennedy, J.E (2003)
- An Assessment of the Evidence for Psychic Functioning — Utts, Jessica (1996)
More in Skeptical
Cognitive Styles and Psi: Psi Researchers Are More Similar to Skeptics Than to Lay Believers
Searching for the Impossible: Parapsychology's Elusive Quest
False-Positive Effect in the Radin Double-Slit Experiment on Observer Consciousness as Determined with the Advanced Meta-Experimental Protocol
Cross-Examining the Case for Precognition: Comment on Mossbridge and Radin (2018)
N,N-Dimethyltryptamine and the Pineal Gland: Separating Fact from Myth
📋 Cite this paper
Hyman, Ray (2010). Meta-Analysis That Conceals More Than It Reveals: Comment on Storm et al. (2010). Psychological Bulletin. https://doi.org/10.1037/a0019676
@article{hyman_2010_meta_analysis_conceals,
title = {Meta-Analysis That Conceals More Than It Reveals: Comment on Storm et al. (2010)},
author = {Hyman, Ray},
year = {2010},
journal = {Psychological Bulletin},
doi = {10.1037/a0019676},
}