Analyzing multimodal communication around a shared tabletop display

Anne Marie Piper*, James D. Hollan

*Corresponding author for this work

Research output: Chapter in Book/Report/Conference proceedingConference contribution

9 Scopus citations


Communication between people is inherently multimodal. People employ speech, facial expressions, eye gaze, and gesture, among other facilities, to support communication and cooperative activity. Complexity of communication increases when a person is without a modality such as hearing, often resulting in dependence on another person or an assistive device to facilitate communication. This paper examines communication about medical topics through Shared Speech Interface, a multimodal tabletop display designed to assist communication between a hearing and deaf individual by converting speech-to-text and representing dialogue history on a shared interactive display surface. We compare communication mediated by a multimodal tabletop display and by a human sign language interpreter. Results indicate that the multimodal tabletop display (1) allows the deaf patient to watch the doctor when she is speaking, (2) encourages the doctor to exploit multimodal communication such as co-occurring gesture-speech, and (3) provides shared access to persistent, collaboratively produced representations of conversation. We also describe extensions of this communication technology, discuss how multimodal analysis techniques are useful in understanding the affects of multiuser multimodal tabletop systems, and briefly allude to the potential of applying computer vision techniques to assist analysis.

Original languageEnglish (US)
Title of host publicationECSCW 2009 - Proceedings of the 11th European Conference on Computer Supported Cooperative Work
Number of pages20
StatePublished - Dec 1 2009
Event11th European Conference on Computer Supported Cooperative Work, ECSCW 2009 - Vienna, Austria
Duration: Sep 7 2009Sep 11 2009


Other11th European Conference on Computer Supported Cooperative Work, ECSCW 2009

ASJC Scopus subject areas

  • Computer Networks and Communications
  • Computer Science Applications


Dive into the research topics of 'Analyzing multimodal communication around a shared tabletop display'. Together they form a unique fingerprint.

Cite this