Consideration of lombard effect for speechreading

Abstract

We propose a method for integrating audio and visual information to enhance speech recognition in adverse environments. We train the audio hidden Markov model and the visual hidden Markov model separately, and then use a Viterbi algorithm to decode both channels in parallel. The decoding process is asynchronous between the two channels to capture the asynchronous nature of audio and visual speech. We test the proposed method using speech corrupted by various types of noises and speech with the Lombard effect.