📌 Appears in:
Plain English Summary
Published in the APA's prestigious Psychological Methods, Kennedy lays down a challenge: if you want your psi study to actually prove something, you need to be able to show your idea is wrong, not just right. That's what "falsifiable" means -- designing experiments where failure is a real possibility. His framework combines several statistical tools and lands on a striking requirement: for the small effects typical in psi research, you'd need roughly 860 to 1,084 participants to run a properly powered study. That's a big ask! He also argues that simply claiming "any tiny effect counts" makes a hypothesis untestable, and that looking back at old studies to build your case is exploratory at best. Instead, researchers should plan ahead and spell out exactly what would count as success, failure, or an unclear result.
Research Notes
Published in APA's Psychological Methods, this formalizes Kennedy's decade-long argument that psi research must adopt falsifiable designs with prespecified minimum effect sizes. Directly extends Kennedy (2016) and cites the Transparent Psi Project and Maier et al. (2020) Bem replication as exemplars. Sets a bar that would require N > 1,000 for most psi paradigms.
Falsifiable research requires that study designs can provide evidence a hypothesis is false as well as true. This article integrates power analysis, equivalence testing, Bayesian operating characteristics, and preregistration into a framework for falsifiable confirmatory research. Power >= .95 for a prespecified minimum effect size is optimal; .90 is good. If any nonzero effect is considered meaningful, the hypothesis is unfalsifiable. For d = 0.20, sample sizes of 858-1,084 are needed at adequate power. The alternative hypothesis can be rejected via noncentral t distributions when power is high. Preregistrations should specify criteria for evidence the hypothesis is true, false, or inconclusive. Retrospective meta-analyses are exploratory; prospective meta-analysis is preferred.
Links
Related Papers
Cites
- Feeling the Future: Experimental Evidence for Anomalous Retroactive Influences on Cognition and Affect — Bem, Daryl J (2011)
- A Preregistered Multi-Lab Replication of Maier et al. (2014, Exp. 4) Testing Retroactive Avoidance — Maier, Markus A (2020)
- Power failure: why small sample size undermines the reliability of neuroscience — Button, Katherine S (2013)
- False-Positive Psychology: Undisclosed Flexibility in Data Collection and Analysis Allows Presenting Anything as Significant — Simmons, Joseph P (2011)
- An Agenda for Purely Confirmatory Research — Wagenmakers, Eric-Jan (2012)
- Why Most Published Research Findings Are False — Ioannidis, John P.A (2005)
- Estimating the Reproducibility of Psychological Science — Open Science Collaboration (2015)
Extends
Same Research Program
- Addressing Researcher Fraud: Retrospective, Real-Time, and Preventive Strategies — Including Legal Points and Data Management That Prevents Fraud — Kennedy, James E (2024)
- Can Parapsychology Move Beyond the Controversies of Retrospective Meta-Analyses? — Kennedy, J.E (2013)
- Bayesian and Classical Hypothesis Testing: Practical Differences for a Controversial Area of Research — Kennedy, J.E (2014)
- Experimenter Fraud: What Are Appropriate Methodological Standards? — Kennedy, J.E (2017)
- Why Is Psi So Elusive? A Review and Proposed Model — Kennedy, James E (2001)
- The Capricious, Actively Evasive, Unsustainable Nature of Psi: A Summary and Hypotheses — Kennedy, J.E (2003)
- Conclusions about Paranormal Phenomena — Kennedy, J.E (2013)
- Is the Methodological Revolution in Psychology Over or Just Beginning? — Kennedy, J.E (2016)
- A Proposal and Challenge for Proponents and Skeptics of Psi — Kennedy, J.E (2004)
- Options for Prospective Meta-Analysis and Introduction of Registration-Based Prospective Meta-Analysis — Watt, Caroline A (2017)
More in Methodology
Paranormal belief, conspiracy endorsement, and positive wellbeing: a network analysis
Quantum Aspects of the Brain-Mind Relationship: A Hypothesis with Supporting Evidence
Paranormal beliefs and cognitive function: A systematic review and assessment of study quality across four decades of research
Experimental evidence of non-classical brain functions
Self-Ascribed Paranormal Ability: Reflexive Thematic Analysis
📋 Cite this paper
Kennedy, James E (2024). Planning Falsifiable Confirmatory Research. Psychological Methods. https://doi.org/10.1037/met0000639
@article{kennedy_2024_falsifiable_research,
title = {Planning Falsifiable Confirmatory Research},
author = {Kennedy, James E},
year = {2024},
journal = {Psychological Methods},
doi = {10.1037/met0000639},
}