A discipline-wide investigation of the replicability of Psychology papers over the past two decades

Wu Youyou, Yang Yang, Brian Uzzi*

*Corresponding author for this work

Research output: Contribution to journalArticlepeer-review

17 Scopus citations

Abstract

Conjecture about the weak replicability in social sciences has made scholars eager to quantify the scale and scope of replication failure for a discipline. Yet small-scale manual replication methods alone are ill-suited to deal with this big data problem. Here, we conduct a discipline-wide replication census in science. Our sample (N = 14,126 papers) covers nearly all papers published in the six top-tier Psychology journals over the past 20 y. Using a validated machine learning model that estimates a paper’s likelihood of replication, we found evidence that both supports and refutes speculations drawn from a relatively small sample of manual replications. First, we find that a single overall replication rate of Psychology poorly captures the varying degree of replicability among subfields. Second, we find that replication rates are strongly correlated with research methods in all subfields. Experiments replicate at a significantly lower rate than do non-experimental studies. Third, we find that authors’ cumulative publication number and citation impact are positively related to the likelihood of replication, while other proxies of research quality and rigor, such as an author’s university prestige and a paper’s citations, are unrelated to replicability. Finally, contrary to the ideal that media attention should cover replicable research, we find that media attention is positively related to the likelihood of replication failure. Our assessments of the scale and scope of replicability are important next steps toward broadly resolving issues of replicability.

Original languageEnglish (US)
Article numbere2208863120
JournalProceedings of the National Academy of Sciences of the United States of America
Volume120
Issue number6
DOIs
StatePublished - Feb 7 2023

Keywords

  • machine learning
  • psychology
  • replication
  • science of science

ASJC Scopus subject areas

  • General

Fingerprint

Dive into the research topics of 'A discipline-wide investigation of the replicability of Psychology papers over the past two decades'. Together they form a unique fingerprint.

Cite this