Computationally Augmented Ethnography: Emotion Tracking and Learning in Museum Games

Kit Martin*, Emily Q. Wang, Connor Bain, Marcelo Worsley

*Corresponding author for this work

Research output: Chapter in Book/Report/Conference proceedingConference contribution

9 Scopus citations


In this paper, we describe a way of using multi-modal learning analytics to augment qualitative data. We extract facial expressions that may indicate particular emotions from videos of dyads playing an interactive table-top game built for a museum. From this data, we explore the correlation between students’ understanding of the biological and complex systems concepts showcased in the learning environment and their facial expressions. First, we show how information retrieval techniques can be used on facial expression features to investigate emotional variation during key moments of the interaction. Second, we connect these features to moments of learning identified by traditional qualitative methods. Finally, we present an initial pilot using these methods in concert to identify key moments in multiple modalities. We end with a discussion of our preliminary findings on interweaving machine and human analytical approaches.

Original languageEnglish (US)
Title of host publicationAdvances in Quantitative Ethnography - 1st International Conference, ICQE 2019, Proceedings
EditorsBrendan Eagan, Amanda Siebert-Evenstone, Morten Misfeldt
Number of pages13
ISBN (Print)9783030332310
StatePublished - 2019
Event1st International Conference on Quantitative Ethnography, ICQE 2019 - Madison, United States
Duration: Oct 20 2019Oct 22 2019

Publication series

NameCommunications in Computer and Information Science
ISSN (Print)1865-0929
ISSN (Electronic)1865-0937


Conference1st International Conference on Quantitative Ethnography, ICQE 2019
Country/TerritoryUnited States


  • Affect tracking
  • Game-based learning
  • Multimodal learning analytics
  • Physical traces

ASJC Scopus subject areas

  • General Computer Science
  • General Mathematics


Dive into the research topics of 'Computationally Augmented Ethnography: Emotion Tracking and Learning in Museum Games'. Together they form a unique fingerprint.

Cite this