Multimodal knowledge capture from text and diagrams

Kate Lockwood*, Kenneth D Forbus

*Corresponding author for this work

Research output: Chapter in Book/Report/Conference proceedingConference contribution

8 Scopus citations


Many information sources use multiple modalities, such as textbooks, which contain both text and diagrams. Each captures information that is hard to express in the other, and evidence suggests that multimodal information leads to better retention and transfer in human learners. This paper describes a system that captures textbook knowledge, using simplified English text and sketched versions of diagrams. We present experimental results showing it can use captured knowledge to answer questions from the textbook's curriculum.

Original languageEnglish (US)
Title of host publicationK-CAP'09 - Proceedings of the 5th International Conference on Knowledge Capture
Number of pages8
StatePublished - Nov 23 2009
Event5th International Conference on Knowledge Capture, K-CAP'09 - Redondo Beach, CA, United States
Duration: Sep 1 2009Sep 4 2009


Other5th International Conference on Knowledge Capture, K-CAP'09
CountryUnited States
CityRedondo Beach, CA


  • Algorithms
  • Experimentation
  • I.2.4 knowledge representation formalisms and methods

ASJC Scopus subject areas

  • Computational Theory and Mathematics
  • Software

Fingerprint Dive into the research topics of 'Multimodal knowledge capture from text and diagrams'. Together they form a unique fingerprint.

Cite this