A Multimodal Analysis of Making

Marcelo Worsley*, Paulo Blikstein

*Corresponding author for this work

Research output: Contribution to journalArticlepeer-review

46 Scopus citations


This paper presents three multimodal learning analytic approaches from a hands-on learning activity. We use video, audio, gesture and bio-physiology data from a two-condition study (N = 20), to identify correlations between the multimodal data, experimental condition, and two learning outcomes: design quality and learning. The three approaches incorporate: 1) human-annotated coding of video data, 2) automated coding of gesture, audio and bio-physiological data and, 3) concatenated human-annotated and automatically annotated data. Within each analysis we employ the same machine learning and sequence mining techniques. Ultimately we find that each approach provides different affordances depending on the similarity metric and the dependent variable. For example, the analysis based on human-annotated data found strong correlations among multimodal behaviors, experimental condition, success and learning, when we relaxed constraints on temporal similarity. The second approach performed well when comparing students’ multimodal behaviors as a time series, but was less effective using the temporally relaxed similarity metric. The take-away is that there are several strategies for doing multimodal learning analytics, and that many of these approaches can provide a meaningful glimpse into a complex data set, glimpses that may be difficult to identify using traditional approaches.

Original languageEnglish (US)
Pages (from-to)385-419
Number of pages35
JournalInternational Journal of Artificial Intelligence in Education
Issue number3
StatePublished - Sep 1 2018


  • Constructionism
  • Learning analytics
  • Signal processing

ASJC Scopus subject areas

  • Education
  • Computational Theory and Mathematics


Dive into the research topics of 'A Multimodal Analysis of Making'. Together they form a unique fingerprint.

Cite this