TY - GEN

T1 - How Visualizing Inferential Uncertainty Can Mislead Readers about Treatment Effects in Scientific Results

AU - Hofman, Jake M.

AU - Goldstein, Daniel G.

AU - Hullman, Jessica

N1 - Publisher Copyright:
© 2020 ACM.
Copyright:
Copyright 2020 Elsevier B.V., All rights reserved.

PY - 2020/4/21

Y1 - 2020/4/21

N2 - When presenting visualizations of experimental results, scientists often choose to display either inferential uncertainty (e.g., uncertainty in the estimate of a population mean) or outcome uncertainty (e.g., variation of outcomes around that mean) about their estimates. How does this choice impact readers' beliefs about the size of treatment effects? We investigate this question in two experiments comparing 95% confidence intervals (means and standard errors) to 95% prediction intervals (means and standard deviations). The first experiment finds that participants are willing to pay more for and overestimate the effect of a treatment when shown confidence intervals relative to prediction intervals. The second experiment evaluates how alternative visualizations compare to standard visualizations for different effect sizes. We find that axis rescaling reduces error, but not as well as prediction intervals or animated hypothetical outcome plots (HOPs), and that depicting inferential uncertainty causes participants to underestimate variability in individual outcomes.

AB - When presenting visualizations of experimental results, scientists often choose to display either inferential uncertainty (e.g., uncertainty in the estimate of a population mean) or outcome uncertainty (e.g., variation of outcomes around that mean) about their estimates. How does this choice impact readers' beliefs about the size of treatment effects? We investigate this question in two experiments comparing 95% confidence intervals (means and standard errors) to 95% prediction intervals (means and standard deviations). The first experiment finds that participants are willing to pay more for and overestimate the effect of a treatment when shown confidence intervals relative to prediction intervals. The second experiment evaluates how alternative visualizations compare to standard visualizations for different effect sizes. We find that axis rescaling reduces error, but not as well as prediction intervals or animated hypothetical outcome plots (HOPs), and that depicting inferential uncertainty causes participants to underestimate variability in individual outcomes.

KW - confidence intervals

KW - effect sizes

KW - judgment and decision making

KW - prediction intervals

KW - uncertainty visualization

UR - http://www.scopus.com/inward/record.url?scp=85091276129&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=85091276129&partnerID=8YFLogxK

U2 - 10.1145/3313831.3376454

DO - 10.1145/3313831.3376454

M3 - Conference contribution

AN - SCOPUS:85091276129

T3 - Conference on Human Factors in Computing Systems - Proceedings

BT - CHI 2020 - Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems

PB - Association for Computing Machinery

T2 - 2020 ACM CHI Conference on Human Factors in Computing Systems, CHI 2020

Y2 - 25 April 2020 through 30 April 2020

ER -