Abstract
We propose a machine learning system that learns to choose human gestures to accompany novel text. The system is trained on scripts comprised of speech and animations that were hand-coded by professional animators and shipped in video games. We treat this as a text-classification problem, classifying speech as corresponding with specific classes of gestures. We have built and tested two separate classifiers. The first is trained simply on the frequencies of different animations in the corpus. The second extracts text features from each script, and maps these features to the gestures that accompany the script. We have experimented with using a number of features of the text, including n-grams, emotional valence of the text, and parts-of-speech. Using a nave Bayes classifier, the system learns to associate these features with appropriate classes of gestures. Once trained, the system can be given novel text for which it will attempt to assign appropriate gestures. We examine the performance of the two classifiers by using n-fold cross-validation over our training data, as well as two user studies of subjective evaluation of the results. Although there are many possible applications of automated gesture assignment, we hope to apply this technique to a system that produces an automated news show.
| Original language | English (US) |
|---|---|
| Title of host publication | Proceedings of the Fifteenth ACM International Conference on Multimedia, MM'07 |
| Pages | 827-830 |
| Number of pages | 4 |
| DOIs | |
| State | Published - Dec 1 2007 |
| Event | 15th ACM International Conference on Multimedia, MM'07 - Augsburg, Bavaria, Germany Duration: Sep 24 2007 → Sep 29 2007 |
Other
| Other | 15th ACM International Conference on Multimedia, MM'07 |
|---|---|
| Country | Germany |
| City | Augsburg, Bavaria |
| Period | 9/24/07 → 9/29/07 |
Fingerprint
Keywords
- Animation
- Gestures
- Machine learning
- Nave bayes
ASJC Scopus subject areas
- Computer Science(all)
Cite this
}
Learning to gesture : Applying appropriate animations to spoken text. / Nichols, Nathan; Liu, Jiahui; Pardo, Bryan A; Hammond, Kristian J; Birnbaum, Lawrence A.
Proceedings of the Fifteenth ACM International Conference on Multimedia, MM'07. 2007. p. 827-830.Research output: Chapter in Book/Report/Conference proceeding › Conference contribution
TY - GEN
T1 - Learning to gesture
T2 - Applying appropriate animations to spoken text
AU - Nichols, Nathan
AU - Liu, Jiahui
AU - Pardo, Bryan A
AU - Hammond, Kristian J
AU - Birnbaum, Lawrence A
PY - 2007/12/1
Y1 - 2007/12/1
N2 - We propose a machine learning system that learns to choose human gestures to accompany novel text. The system is trained on scripts comprised of speech and animations that were hand-coded by professional animators and shipped in video games. We treat this as a text-classification problem, classifying speech as corresponding with specific classes of gestures. We have built and tested two separate classifiers. The first is trained simply on the frequencies of different animations in the corpus. The second extracts text features from each script, and maps these features to the gestures that accompany the script. We have experimented with using a number of features of the text, including n-grams, emotional valence of the text, and parts-of-speech. Using a nave Bayes classifier, the system learns to associate these features with appropriate classes of gestures. Once trained, the system can be given novel text for which it will attempt to assign appropriate gestures. We examine the performance of the two classifiers by using n-fold cross-validation over our training data, as well as two user studies of subjective evaluation of the results. Although there are many possible applications of automated gesture assignment, we hope to apply this technique to a system that produces an automated news show.
AB - We propose a machine learning system that learns to choose human gestures to accompany novel text. The system is trained on scripts comprised of speech and animations that were hand-coded by professional animators and shipped in video games. We treat this as a text-classification problem, classifying speech as corresponding with specific classes of gestures. We have built and tested two separate classifiers. The first is trained simply on the frequencies of different animations in the corpus. The second extracts text features from each script, and maps these features to the gestures that accompany the script. We have experimented with using a number of features of the text, including n-grams, emotional valence of the text, and parts-of-speech. Using a nave Bayes classifier, the system learns to associate these features with appropriate classes of gestures. Once trained, the system can be given novel text for which it will attempt to assign appropriate gestures. We examine the performance of the two classifiers by using n-fold cross-validation over our training data, as well as two user studies of subjective evaluation of the results. Although there are many possible applications of automated gesture assignment, we hope to apply this technique to a system that produces an automated news show.
KW - Animation
KW - Gestures
KW - Machine learning
KW - Nave bayes
UR - http://www.scopus.com/inward/record.url?scp=37849024055&partnerID=8YFLogxK
UR - http://www.scopus.com/inward/citedby.url?scp=37849024055&partnerID=8YFLogxK
U2 - 10.1145/1291233.1291421
DO - 10.1145/1291233.1291421
M3 - Conference contribution
SN - 9781595937025
SP - 827
EP - 830
BT - Proceedings of the Fifteenth ACM International Conference on Multimedia, MM'07
ER -