Learning linkages: Integrating data streams of multiple modalities and timescales

Ran Liu, John Stamper, Jodi Davenport, Scott Crossley, Danielle McNamara, Kalonji Nzinga, Bruce L Sherin

Research output: Contribution to journalArticle

Abstract

Increasingly, student work is being conducted on computers and online, producing vast amounts of learning-related data. The educational analytics fields have produced many insights about learning based solely on tutoring systems' automatically logged data, or “log data.” But log data leave out important contextual information about the learning experience. For example, a student working at a computer might be working independently with few outside influences. Alternatively, he or she might be in a lively classroom, with other students around, talking and offering suggestions. Tools that capture these other experiences have potential to augment and complement log data. However, the collection of rich, multimodal data streams and the increased complexity and heterogeneity in the resulting data pose many challenges to researchers. Here, we present two empirical studies that take advantage of multimodal data sources to enrich our understanding of student learning. We leverage and extend quantitative models of student learning to incorporate insights derived jointly from data collected in multiple modalities (log data, video, and high-fidelity audio) and contexts (individual vs. collaborative classroom learning). We discuss the unique benefits of multimodal data and present methods that take advantage of such benefits while easing the burden on researchers' time and effort.

Original languageEnglish (US)
Pages (from-to)99-109
Number of pages11
JournalJournal of Computer Assisted Learning
Volume35
Issue number1
DOIs
StatePublished - Feb 1 2019

Fingerprint

Students
learning
student
classroom
experience
video

Keywords

  • STEM education
  • collaborative learning
  • intelligent tutoring systems
  • log data
  • multi-modal data analytics
  • natural language processing

ASJC Scopus subject areas

  • Education
  • Computer Science Applications

Cite this

Liu, Ran ; Stamper, John ; Davenport, Jodi ; Crossley, Scott ; McNamara, Danielle ; Nzinga, Kalonji ; Sherin, Bruce L. / Learning linkages : Integrating data streams of multiple modalities and timescales. In: Journal of Computer Assisted Learning. 2019 ; Vol. 35, No. 1. pp. 99-109.
@article{6c325e705102443ab5f9d1a3ec84e95b,
title = "Learning linkages: Integrating data streams of multiple modalities and timescales",
abstract = "Increasingly, student work is being conducted on computers and online, producing vast amounts of learning-related data. The educational analytics fields have produced many insights about learning based solely on tutoring systems' automatically logged data, or “log data.” But log data leave out important contextual information about the learning experience. For example, a student working at a computer might be working independently with few outside influences. Alternatively, he or she might be in a lively classroom, with other students around, talking and offering suggestions. Tools that capture these other experiences have potential to augment and complement log data. However, the collection of rich, multimodal data streams and the increased complexity and heterogeneity in the resulting data pose many challenges to researchers. Here, we present two empirical studies that take advantage of multimodal data sources to enrich our understanding of student learning. We leverage and extend quantitative models of student learning to incorporate insights derived jointly from data collected in multiple modalities (log data, video, and high-fidelity audio) and contexts (individual vs. collaborative classroom learning). We discuss the unique benefits of multimodal data and present methods that take advantage of such benefits while easing the burden on researchers' time and effort.",
keywords = "STEM education, collaborative learning, intelligent tutoring systems, log data, multi-modal data analytics, natural language processing",
author = "Ran Liu and John Stamper and Jodi Davenport and Scott Crossley and Danielle McNamara and Kalonji Nzinga and Sherin, {Bruce L}",
year = "2019",
month = "2",
day = "1",
doi = "10.1111/jcal.12315",
language = "English (US)",
volume = "35",
pages = "99--109",
journal = "Journal of Computer Assisted Learning",
issn = "0266-4909",
publisher = "Wiley-Blackwell",
number = "1",

}

Learning linkages : Integrating data streams of multiple modalities and timescales. / Liu, Ran; Stamper, John; Davenport, Jodi; Crossley, Scott; McNamara, Danielle; Nzinga, Kalonji; Sherin, Bruce L.

In: Journal of Computer Assisted Learning, Vol. 35, No. 1, 01.02.2019, p. 99-109.

Research output: Contribution to journalArticle

TY - JOUR

T1 - Learning linkages

T2 - Integrating data streams of multiple modalities and timescales

AU - Liu, Ran

AU - Stamper, John

AU - Davenport, Jodi

AU - Crossley, Scott

AU - McNamara, Danielle

AU - Nzinga, Kalonji

AU - Sherin, Bruce L

PY - 2019/2/1

Y1 - 2019/2/1

N2 - Increasingly, student work is being conducted on computers and online, producing vast amounts of learning-related data. The educational analytics fields have produced many insights about learning based solely on tutoring systems' automatically logged data, or “log data.” But log data leave out important contextual information about the learning experience. For example, a student working at a computer might be working independently with few outside influences. Alternatively, he or she might be in a lively classroom, with other students around, talking and offering suggestions. Tools that capture these other experiences have potential to augment and complement log data. However, the collection of rich, multimodal data streams and the increased complexity and heterogeneity in the resulting data pose many challenges to researchers. Here, we present two empirical studies that take advantage of multimodal data sources to enrich our understanding of student learning. We leverage and extend quantitative models of student learning to incorporate insights derived jointly from data collected in multiple modalities (log data, video, and high-fidelity audio) and contexts (individual vs. collaborative classroom learning). We discuss the unique benefits of multimodal data and present methods that take advantage of such benefits while easing the burden on researchers' time and effort.

AB - Increasingly, student work is being conducted on computers and online, producing vast amounts of learning-related data. The educational analytics fields have produced many insights about learning based solely on tutoring systems' automatically logged data, or “log data.” But log data leave out important contextual information about the learning experience. For example, a student working at a computer might be working independently with few outside influences. Alternatively, he or she might be in a lively classroom, with other students around, talking and offering suggestions. Tools that capture these other experiences have potential to augment and complement log data. However, the collection of rich, multimodal data streams and the increased complexity and heterogeneity in the resulting data pose many challenges to researchers. Here, we present two empirical studies that take advantage of multimodal data sources to enrich our understanding of student learning. We leverage and extend quantitative models of student learning to incorporate insights derived jointly from data collected in multiple modalities (log data, video, and high-fidelity audio) and contexts (individual vs. collaborative classroom learning). We discuss the unique benefits of multimodal data and present methods that take advantage of such benefits while easing the burden on researchers' time and effort.

KW - STEM education

KW - collaborative learning

KW - intelligent tutoring systems

KW - log data

KW - multi-modal data analytics

KW - natural language processing

UR - http://www.scopus.com/inward/record.url?scp=85055712339&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=85055712339&partnerID=8YFLogxK

U2 - 10.1111/jcal.12315

DO - 10.1111/jcal.12315

M3 - Article

VL - 35

SP - 99

EP - 109

JO - Journal of Computer Assisted Learning

JF - Journal of Computer Assisted Learning

SN - 0266-4909

IS - 1

ER -