Multicraft: A multimodal interface for supporting and studying learning in minecraft

Marcelo Worsley*, Kevin Mendoza Tudares, Timothy Mwiti, Mitchell Zhen, Marc Jiang

*Corresponding author for this work

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Abstract

In this paper, we present work on bringing multimodal interaction to Minecraft. The platform, Multicraft, incorporates speech-based input, eye tracking, and natural language understanding to facilitate more equitable gameplay in Minecraft. We tested the platform with elementary, middle school students and college students through a collection of studies. Students found each of the provided modalities to be a compelling way to play Minecraft. Additionally, we discuss the ways that these different types of multimodal data can be used to identify the meaningful spatial reasoning practices that students demonstrate while playing Minecraft. Collectively, this paper emphasizes the opportunity to bridge a multimodal interface with a means for collecting rich data that can better support diverse learners in non-traditional learning environments.

Original languageEnglish (US)
Title of host publicationHCI in Games
Subtitle of host publicationSerious and Immersive Games - 3rd International Conference, HCI-Games 2021, Held as Part of the 23rd HCI International Conference, HCII 2021, Proceedings
EditorsXiaowen Fang
PublisherSpringer Science and Business Media Deutschland GmbH
Pages113-131
Number of pages19
ISBN (Print)9783030774134
DOIs
StatePublished - 2021
Event3rd International Conference on HCI in Games, HCI-Games 2021, held as part of the 23rd International Conference, HCI International 2021 - Virtual, Online
Duration: Jul 24 2021Jul 29 2021

Publication series

NameLecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)
Volume12790 LNCS
ISSN (Print)0302-9743
ISSN (Electronic)1611-3349

Conference

Conference3rd International Conference on HCI in Games, HCI-Games 2021, held as part of the 23rd International Conference, HCI International 2021
CityVirtual, Online
Period7/24/217/29/21

Keywords

  • Constructionism
  • Data mining
  • Games
  • Spatial reasoning

ASJC Scopus subject areas

  • Theoretical Computer Science
  • General Computer Science

Fingerprint

Dive into the research topics of 'Multicraft: A multimodal interface for supporting and studying learning in minecraft'. Together they form a unique fingerprint.

Cite this