In Pursuit of Error: A Survey of Uncertainty Visualization Evaluation

Jessica Ruth Hullman, Xiaoli Qiao, Michael Correll, Alex Kale, Matthew Kay

Research output: Contribution to journalReview article

13 Scopus citations

Abstract

Understanding and accounting for uncertainty is critical to effectively reasoning about visualized data. However, evaluating the impact of an uncertainty visualization is complex due to the difficulties that people have interpreting uncertainty and the challenge of defining correct behavior with uncertainty information. Currently, evaluators of uncertainty visualization must rely on general purpose visualization evaluation frameworks which can be ill-equipped to provide guidance with the unique difficulties of assessing judgments under uncertainty. To help evaluators navigate these complexities, we present a taxonomy for characterizing decisions made in designing an evaluation of an uncertainty visualization. Our taxonomy differentiates six levels of decisions that comprise an uncertainty visualization evaluation: the behavioral targets of the study, expected effects from an uncertainty visualization, evaluation goals, measures, elicitation techniques, and analysis approaches. Applying our taxonomy to 86 user studies of uncertainty visualizations, we find that existing evaluation practice, particularly in visualization research, focuses on Performance and Satisfaction-based measures that assume more predictable and statistically-driven judgment behavior than is suggested by research on human judgment and decision making. We reflect on common themes in evaluation practice concerning the interpretation and semantics of uncertainty, the use of confidence reporting, and a bias toward evaluating performance as accuracy rather than decision quality. We conclude with a concrete set of recommendations for evaluators designed to reduce the mismatch between the conceptualization of uncertainty in visualization versus other fields.

Original languageEnglish (US)
Article number8457476
Pages (from-to)903-913
Number of pages11
JournalIEEE Transactions on Visualization and Computer Graphics
Volume25
Issue number1
DOIs
StatePublished - Jan 2019

Keywords

  • Uncertainty visualization
  • probability distribution
  • subjective confidence
  • user study

ASJC Scopus subject areas

  • Software
  • Signal Processing
  • Computer Vision and Pattern Recognition
  • Computer Graphics and Computer-Aided Design

Fingerprint Dive into the research topics of 'In Pursuit of Error: A Survey of Uncertainty Visualization Evaluation'. Together they form a unique fingerprint.

  • Cite this