X Goes First: Teaching a Simple Game through Multimodal Interaction

T. Hinrichs, K. Forbus

Research output: Contribution to journalArticlepeer-review


What would it take to teach a computer to play a game entirely through language and sketching? In this paper, we present an implemented program through which an instructor can teach the rules of simple board games using such input. We describe the architecture, information flow, and vocabulary of instructional events and walk through an annotated example. In our approach, the instructional and communication events guide abductive reasoning for language interpretation and help to integrate information from sketching and language. Having a general target representation enables the learning process to be viewed more as translation and problem solving than as induction. Lastly, learning by demonstration complements and extends instruction, resulting in concrete, operational rules.
Original languageEnglish
Pages (from-to)31-46
JournalAdvances in Cognitive Systems
StatePublished - 2014


Dive into the research topics of 'X Goes First: Teaching a Simple Game through Multimodal Interaction'. Together they form a unique fingerprint.

Cite this