Lessons from the First Two Years of Operating a Study Registry
🧐 Skeptical/Critical📌 Appears in:
Plain English Summary
Back in 2012, right when psychology was reckoning with its massive replication crisis, the Koestler Parapsychology Unit at the University of Edinburgh launched a study registry -- basically a public promise board where researchers had to declare exactly what they planned to test before they tested it. The idea was borrowed from clinical medicine, where pre-registration keeps drug companies honest. The results were eye-opening: nearly every submission had gaps or missing details. Even more damning, the authors called out the popular Open Science Framework for letting researchers quietly hide their registrations after peeking at results -- completely defeating the purpose. They also pointed to famous parapsychology studies by Bem and others that, despite big splashes, never settled the debates they sparked, precisely because confirmatory rigor (locking in your predictions ahead of time) was missing.
Research Notes
Foundational paper on pre-registration in parapsychology. Extends Kennedy (2004) advocacy for formal confirmatory research, which gained traction after the 2012 replication crisis. Practical critique of OSF registration model remains relevant. Directly discusses Bem (2011), Bosch (2006), and Storm (2010) as examples of high-profile parapsychology publications that did not resolve debates.
Opinion article on the first two years of the Koestler Parapsychology Unit study registry at the University of Edinburgh (opened fall 2012). Modeled on clinical trial standards (ICMJE 2005), the registry requires public, prospective, irreversibly public registration with independent review. Key recommendations: classify each hypothesis as exploratory or confirmatory; pre-specify all analysis decisions for confirmatory research; independently review registrations for completeness. Virtually all initial submissions had deficiencies. Critiques OSF registration for allowing experimenters to keep or revert registrations to private after viewing results, undermining the file-drawer protection that registration is meant to provide.
Links
Related Papers
Companion
- Bayesian and Classical Hypothesis Testing: Practical Differences for a Controversial Area of Research — Kennedy, J.E (2014)
- Commentary: Reproducibility in Psychological Science: When Do Psychological Phenomena Exist? — Heino, Matti T. J (2017)
- Addressing Researcher Fraud: Retrospective, Real-Time, and Preventive Strategies — Including Legal Points and Data Management That Prevents Fraud — Kennedy, James E (2024)
Discusses
- An Agenda for Purely Confirmatory Research — Wagenmakers, Eric-Jan (2012)
- Feeling the Future: Experimental Evidence for Anomalous Retroactive Influences on Cognition and Affect — Bem, Daryl J (2011)
- Examining Psychokinesis: The Interaction of Human Intention With Random Number Generators—A Meta-Analysis — Bösch, Holger (2006)
- Meta-Analysis of Free-Response Studies, 1992–2008: Assessing the Noise Reduction Model in Parapsychology — Storm, Lance (2010)
Also by these authors
Options for Prospective Meta-Analysis and Introduction of Registration-Based Prospective Meta-Analysis
There Is Nothing Paranormal about Near-Death Experiences: How Neuroscience Can Explain Seeing Bright Lights, Meeting the Dead, or Being Convinced You Are One of Them
Of Two Minds: Sceptic-Proponent Collaboration within Parapsychology
More in Methodology
Paranormal belief, conspiracy endorsement, and positive wellbeing: a network analysis
Planning Falsifiable Confirmatory Research
Quantum Aspects of the Brain-Mind Relationship: A Hypothesis with Supporting Evidence
Paranormal beliefs and cognitive function: A systematic review and assessment of study quality across four decades of research
Experimental evidence of non-classical brain functions
📋 Cite this paper
Watt, Caroline, Kennedy, James E (2015). Lessons from the First Two Years of Operating a Study Registry. Frontiers in Psychology. https://doi.org/10.3389/fpsyg.2015.00173
@article{watt_2015_lessons,
title = {Lessons from the First Two Years of Operating a Study Registry},
author = {Watt, Caroline and Kennedy, James E},
year = {2015},
journal = {Frontiers in Psychology},
doi = {10.3389/fpsyg.2015.00173},
}