How Visualizing Inferential Uncertainty Can Mislead Readers about Treatment Effects in Scientific Results

Jake M. Hofman, Daniel G. Goldstein, Jessica Hullman

Research output: Chapter in Book/Report/Conference proceedingConference contribution

3 Scopus citations

Abstract

When presenting visualizations of experimental results, scientists often choose to display either inferential uncertainty (e.g., uncertainty in the estimate of a population mean) or outcome uncertainty (e.g., variation of outcomes around that mean) about their estimates. How does this choice impact readers' beliefs about the size of treatment effects? We investigate this question in two experiments comparing 95% confidence intervals (means and standard errors) to 95% prediction intervals (means and standard deviations). The first experiment finds that participants are willing to pay more for and overestimate the effect of a treatment when shown confidence intervals relative to prediction intervals. The second experiment evaluates how alternative visualizations compare to standard visualizations for different effect sizes. We find that axis rescaling reduces error, but not as well as prediction intervals or animated hypothetical outcome plots (HOPs), and that depicting inferential uncertainty causes participants to underestimate variability in individual outcomes.

Original languageEnglish (US)
Title of host publicationCHI 2020 - Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems
PublisherAssociation for Computing Machinery
ISBN (Electronic)9781450367080
DOIs
StatePublished - Apr 21 2020
Event2020 ACM CHI Conference on Human Factors in Computing Systems, CHI 2020 - Honolulu, United States
Duration: Apr 25 2020Apr 30 2020

Publication series

NameConference on Human Factors in Computing Systems - Proceedings

Conference

Conference2020 ACM CHI Conference on Human Factors in Computing Systems, CHI 2020
CountryUnited States
CityHonolulu
Period4/25/204/30/20

Keywords

  • confidence intervals
  • effect sizes
  • judgment and decision making
  • prediction intervals
  • uncertainty visualization

ASJC Scopus subject areas

  • Computer Graphics and Computer-Aided Design
  • Human-Computer Interaction
  • Software

Fingerprint Dive into the research topics of 'How Visualizing Inferential Uncertainty Can Mislead Readers about Treatment Effects in Scientific Results'. Together they form a unique fingerprint.

Cite this