Audio-visual anticipatory coarticulation modeling by human and machin

Louis H. Terry*, Karen Livescu, Janet B. Pierrehumbert, Aggelos K Katsaggelos

*Corresponding author for this work

Research output: Contribution to conferencePaper

3 Scopus citations

Abstract

The phenomenon of anticipatory coarticulation provides a basis for the observed asynchrony between the acoustic and visual onsets of phones in certain linguistic contexts. This type of asynchrony is typically not explicitly modeled in audio-visual speech models. In this work, we study within-word audiovisual asynchrony using manual labels of words in which theory suggests that audio-visual asynchrony should occur, and show that these hand labels confirm the theory. We then introduce a new statistical model of audio-visual speech, the asynchrony-dependent transition (ADT) model. This model allows asynchrony between audio and video states within word boundaries, where the audio and video state transitions depend not only on the state of that modality, but also on the instantaneous asynchrony. The ADT model outperforms a baseline synchronous model in mimicking the hand labels in a forced alignment task, and its behavior as parameters are changed conforms to our expectations about anticipatory coarticulation. The same model could be used for speech recognition, although here we consider it only for the task of forced alignment for linguistic analysis.

Original languageEnglish (US)
Pages2682-2685
Number of pages4
StatePublished - Dec 1 2010
Event11th Annual Conference of the International Speech Communication Association: Spoken Language Processing for All, INTERSPEECH 2010 - Makuhari, Chiba, Japan
Duration: Sep 26 2010Sep 30 2010

Other

Other11th Annual Conference of the International Speech Communication Association: Spoken Language Processing for All, INTERSPEECH 2010
CountryJapan
CityMakuhari, Chiba
Period9/26/109/30/10

Keywords

  • Anticipatory coarticulation
  • Audio-visual asynchrony
  • Audio-visual speech recognition
  • Dynamic Bayesian networks

ASJC Scopus subject areas

  • Language and Linguistics
  • Speech and Hearing

Fingerprint Dive into the research topics of 'Audio-visual anticipatory coarticulation modeling by human and machin'. Together they form a unique fingerprint.

  • Cite this

    Terry, L. H., Livescu, K., Pierrehumbert, J. B., & Katsaggelos, A. K. (2010). Audio-visual anticipatory coarticulation modeling by human and machin. 2682-2685. Paper presented at 11th Annual Conference of the International Speech Communication Association: Spoken Language Processing for All, INTERSPEECH 2010, Makuhari, Chiba, Japan.