How good is 85%? A survey tool to connect classifier evaluation to acceptability of accuracy

Matthew Kay, Shwetak N. Patel, Julie A. Kientz

Research output: Chapter in Book/Report/Conference proceedingConference contribution

32 Scopus citations

Abstract

Many HCI and ubiquitous computing systems are characterized by two important properties: Their output is uncertain- It has an associated accuracy that researchers attempt to optimize-and this uncertainty is user-facing-it directly affects the quality of the user experience. Novel classifiers are typically evaluated using measures like the F1 score- But given an F-score of (e.g.) 0.85, how do we know whether this performance is good enough? Is this level of uncertainty actually tolerable to users of the intended application- And do people weight precision and recall equally? We set out to develop a survey instrument that can systematically answer such questions. We introduce a new measure, acceptability of accuracy, and show how to predict it based on measures of classifier accuracy. Out tool allows us to systematically select an objective function to optimize during classifier evaluation, but can also offer new insights into how to design feedback for user-facing classification systems (e.g., by combining a seemingly-low-performing classifier with appropriate feedback to make a highly usable system). It also reveals potential issues with the ubiquitous F1-measure as applied to user-facing systems.

Original languageEnglish (US)
Title of host publicationCHI 2015 - Proceedings of the 33rd Annual CHI Conference on Human Factors in Computing Systems
Subtitle of host publicationCrossings
PublisherAssociation for Computing Machinery
Pages347-356
Number of pages10
ISBN (Electronic)9781450331456
DOIs
StatePublished - Apr 18 2015
Externally publishedYes
Event33rd Annual CHI Conference on Human Factors in Computing Systems, CHI 2015 - Seoul, Korea, Republic of
Duration: Apr 18 2015Apr 23 2015

Publication series

NameConference on Human Factors in Computing Systems - Proceedings
Volume2015-April

Other

Other33rd Annual CHI Conference on Human Factors in Computing Systems, CHI 2015
Country/TerritoryKorea, Republic of
CitySeoul
Period4/18/154/23/15

Keywords

  • Accuracy
  • Accuracy acceptability
  • Classifiers
  • Inference
  • Machine learning
  • Sensors

ASJC Scopus subject areas

  • Software
  • Human-Computer Interaction
  • Computer Graphics and Computer-Aided Design

Fingerprint

Dive into the research topics of 'How good is 85%? A survey tool to connect classifier evaluation to acceptability of accuracy'. Together they form a unique fingerprint.

Cite this