Measuring the Prevalence of Questionable Research Practices With Incentives for Truth Telling
🧐 Skeptical/Critical📌 Appears in:
Plain English Summary
This landmark survey asked how psychology research actually gets done. Nearly 6,000 psychologists were surveyed using Bayesian Truth Serum, a clever method rewarding honest answers. Results were jaw-dropping: 94% admitted at least one shady shortcut. Two-thirds didn't report everything they measured, over half peeked at results before collecting more data, and half cherry-picked which studies to publish. Roughly 1 in 10 appears to have falsified data. These habits aren't slip-ups — they're the norm. This is a cornerstone of the "replication crisis" debate, cutting both ways: the same shortcuts producing false positives could equally undermine failed replications.
Research Notes
Landmark survey empirically grounding the replication crisis. Directly relevant to psi: skeptics cite it as explaining psi’s failure to replicate via QRP contamination; psi proponents note QRPs equally impair skeptical failed-replication studies. Cited by Kennedy papers in this library.
Survey of 5,964 academic psychologists (N=2,155 respondents, 36% response rate) measured prevalence of questionable research practices (QRPs) using Bayesian Truth Serum (BTS) incentives for truthful disclosure. Admissions were surprisingly high: 94% of BTS respondents admitted at least one QRP, including failing to report all dependent measures (66.5%), collecting more data after checking significance (58%), and selectively reporting studies (50%). BTS incentives raised admissions most for less-defensible practices. Geometric-mean estimates suggest ~1 in 10 psychologists has falsified data. Items formed approximate Guttman scale (reproducibility=0.80). Findings suggest QRPs may constitute the de facto scientific norm, with researchers rationalizing borderline behaviors as defensible.
Links
Related Papers
Cites
Companion
- Why Psychologists Must Change the Way They Analyze Their Data: The Case of Psi — Wagenmakers, Eric-Jan (2011)
- Registered Reports: A Method to Increase the Credibility of Published Results — Nosek, Brian A (2014)
- Estimating the Reproducibility of Psychological Science — Open Science Collaboration (2015)
- Commentary: Reproducibility in Psychological Science: When Do Psychological Phenomena Exist? — Heino, Matti T. J (2017)
- Replication Unreliability in Psychology: Elusive Phenomena or "Elusive" Statistical Power? — Tressoldi, Patrizio E (2012)
More in Methodology
Paranormal belief, conspiracy endorsement, and positive wellbeing: a network analysis
Planning Falsifiable Confirmatory Research
Addressing Researcher Fraud: Retrospective, Real-Time, and Preventive Strategies — Including Legal Points and Data Management That Prevents Fraud
Quantum Aspects of the Brain-Mind Relationship: A Hypothesis with Supporting Evidence
Paranormal beliefs and cognitive function: A systematic review and assessment of study quality across four decades of research
📋 Cite this paper
John, Leslie K, Loewenstein, George, Prelec, Drazen (2012). Measuring the Prevalence of Questionable Research Practices With Incentives for Truth Telling. Psychological Science. https://doi.org/10.1177/0956797611430953
@article{john_2012_questionable_practices,
title = {Measuring the Prevalence of Questionable Research Practices With Incentives for Truth Telling},
author = {John, Leslie K and Loewenstein, George and Prelec, Drazen},
year = {2012},
journal = {Psychological Science},
doi = {10.1177/0956797611430953},
}