Comparison of low- and high-level visual features for audio-visual continuous automatic speech recognition

Petar S. Aleksic*, Aggelos K. Katsaggelos

*Corresponding author for this work

Research output: Contribution to journalConference articlepeer-review

20 Scopus citations


In this paper, we compare two different groups of visual features that can be used in addition to audio to improve automatic speech recognition (ASR), high- and low-level visual features. Facial Animation Parameters (FAPs), supported by the MPEG-4 standard for the visual representation of speech are used as high-level visual features in this work. Principal component analysis (PCA) based projection weights of the intensity images of the mouth area were used as low-level visual features. PCA was also applied on the FAPs. We developed an audio-visual ASR (AV-ASR) system and compared its performance for two different visual feature groups, following two approaches, The first approach assumes the same dimensionality for both high- and low-level visual features, while in the second approach the percentage of statistical variance described by the visual features used was the same. Multi-stream Hidden Markov Models (HMMs) and a late integration approach were used to integrate audio and visual information and perform continuous AV-ASR experiments. Experiments were performed at various SNRs (0-30dB) with additive white Gaussian noise on a relatively large vocabulary (approximately 1000 words) database. Conclusions were drawn on the trade off between the dimensionality of the visual features and the amount of speechreading information contained in them and its influence on the AV-ASR performance.

Original languageEnglish (US)
Pages (from-to)V-917-V-920
JournalICASSP, IEEE International Conference on Acoustics, Speech and Signal Processing - Proceedings
StatePublished - 2004
EventProceedings - IEEE International Conference on Acoustics, Speech, and Signal Processing - Montreal, Que, Canada
Duration: May 17 2004May 21 2004

ASJC Scopus subject areas

  • Software
  • Signal Processing
  • Electrical and Electronic Engineering


Dive into the research topics of 'Comparison of low- and high-level visual features for audio-visual continuous automatic speech recognition'. Together they form a unique fingerprint.

Cite this