TY - JOUR
T1 - A Multimodal Analysis of Making
AU - Worsley, Marcelo
AU - Blikstein, Paulo
N1 - Publisher Copyright:
© 2017, International Artificial Intelligence in Education Society.
PY - 2018/9/1
Y1 - 2018/9/1
N2 - This paper presents three multimodal learning analytic approaches from a hands-on learning activity. We use video, audio, gesture and bio-physiology data from a two-condition study (N = 20), to identify correlations between the multimodal data, experimental condition, and two learning outcomes: design quality and learning. The three approaches incorporate: 1) human-annotated coding of video data, 2) automated coding of gesture, audio and bio-physiological data and, 3) concatenated human-annotated and automatically annotated data. Within each analysis we employ the same machine learning and sequence mining techniques. Ultimately we find that each approach provides different affordances depending on the similarity metric and the dependent variable. For example, the analysis based on human-annotated data found strong correlations among multimodal behaviors, experimental condition, success and learning, when we relaxed constraints on temporal similarity. The second approach performed well when comparing students’ multimodal behaviors as a time series, but was less effective using the temporally relaxed similarity metric. The take-away is that there are several strategies for doing multimodal learning analytics, and that many of these approaches can provide a meaningful glimpse into a complex data set, glimpses that may be difficult to identify using traditional approaches.
AB - This paper presents three multimodal learning analytic approaches from a hands-on learning activity. We use video, audio, gesture and bio-physiology data from a two-condition study (N = 20), to identify correlations between the multimodal data, experimental condition, and two learning outcomes: design quality and learning. The three approaches incorporate: 1) human-annotated coding of video data, 2) automated coding of gesture, audio and bio-physiological data and, 3) concatenated human-annotated and automatically annotated data. Within each analysis we employ the same machine learning and sequence mining techniques. Ultimately we find that each approach provides different affordances depending on the similarity metric and the dependent variable. For example, the analysis based on human-annotated data found strong correlations among multimodal behaviors, experimental condition, success and learning, when we relaxed constraints on temporal similarity. The second approach performed well when comparing students’ multimodal behaviors as a time series, but was less effective using the temporally relaxed similarity metric. The take-away is that there are several strategies for doing multimodal learning analytics, and that many of these approaches can provide a meaningful glimpse into a complex data set, glimpses that may be difficult to identify using traditional approaches.
KW - Constructionism
KW - Learning analytics
KW - Signal processing
UR - http://www.scopus.com/inward/record.url?scp=85050918225&partnerID=8YFLogxK
UR - http://www.scopus.com/inward/citedby.url?scp=85050918225&partnerID=8YFLogxK
U2 - 10.1007/s40593-017-0160-1
DO - 10.1007/s40593-017-0160-1
M3 - Article
AN - SCOPUS:85050918225
VL - 28
SP - 385
EP - 419
JO - International Journal of Artificial Intelligence in Education
JF - International Journal of Artificial Intelligence in Education
SN - 1560-4292
IS - 3
ER -