Statistical Significance and the Dichotomization of Evidence

Blakeley B. McShane*, David Gal

*Corresponding author for this work

Research output: Contribution to journalComment/debate

36 Scopus citations

Abstract

In light of recent concerns about reproducibility and replicability, the ASA issued a Statement on Statistical Significance and p-values aimed at those who are not primarily statisticians. While the ASA Statement notes that statistical significance and p-values are “commonly misused and misinterpreted,” it does not discuss and document broader implications of these errors for the interpretation of evidence. In this article, we review research on how applied researchers who are not primarily statisticians misuse and misinterpret p-values in practice and how this can lead to errors in the interpretation of evidence. We also present new data showing, perhaps surprisingly, that researchers who are primarily statisticians are also prone to misuse and misinterpret p-values thus resulting in similar errors. In particular, we show that statisticians tend to interpret evidence dichotomously based on whether or not a p-value crosses the conventional 0.05 threshold for statistical significance. We discuss implications and offer recommendations.

Original languageEnglish (US)
Pages (from-to)885-895
Number of pages11
JournalJournal of the American Statistical Association
Volume112
Issue number519
DOIs
StatePublished - Jul 3 2017

Keywords

  • Null hypothesis significance testing
  • Sociology of science
  • Statistical significance
  • p-value

ASJC Scopus subject areas

  • Statistics and Probability
  • Statistics, Probability and Uncertainty

Fingerprint Dive into the research topics of 'Statistical Significance and the Dichotomization of Evidence'. Together they form a unique fingerprint.

  • Cite this