Learning to gesture: Applying appropriate animations to spoken text

Nathan Nichols*, Jiahui Liu, Bryan A Pardo, Kristian J Hammond, Lawrence A Birnbaum

*Corresponding author for this work

Research output: Chapter in Book/Report/Conference proceedingConference contribution


We propose a machine learning system that learns to choose human gestures to accompany novel text. The system is trained on scripts comprised of speech and animations that were hand-coded by professional animators and shipped in video games. We treat this as a text-classification problem, classifying speech as corresponding with specific classes of gestures. We have built and tested two separate classifiers. The first is trained simply on the frequencies of different animations in the corpus. The second extracts text features from each script, and maps these features to the gestures that accompany the script. We have experimented with using a number of features of the text, including n-grams, emotional valence of the text, and parts-of-speech. Using a nave Bayes classifier, the system learns to associate these features with appropriate classes of gestures. Once trained, the system can be given novel text for which it will attempt to assign appropriate gestures. We examine the performance of the two classifiers by using n-fold cross-validation over our training data, as well as two user studies of subjective evaluation of the results. Although there are many possible applications of automated gesture assignment, we hope to apply this technique to a system that produces an automated news show.

Original languageEnglish (US)
Title of host publicationProceedings of the Fifteenth ACM International Conference on Multimedia, MM'07
Number of pages4
StatePublished - 2007
Event15th ACM International Conference on Multimedia, MM'07 - Augsburg, Bavaria, Germany
Duration: Sep 24 2007Sep 29 2007

Publication series

NameProceedings of the ACM International Multimedia Conference and Exhibition


Other15th ACM International Conference on Multimedia, MM'07
CityAugsburg, Bavaria


  • Animation
  • Gestures
  • Machine learning
  • Nave bayes

ASJC Scopus subject areas

  • General Computer Science


Dive into the research topics of 'Learning to gesture: Applying appropriate animations to spoken text'. Together they form a unique fingerprint.

Cite this