Hidden Markov model based visual speech recognition

Abstract

Speech recognition can be made more accurate if visual speech information such as the movement of the lips is taken into consideration. In this thesis, studies on visual speech processing are presented. Classifiers based on Hidden Markov Model (HMM) are first explored for modeling and identifying the basic visual speech elements. Considering that the temporal features of visual speech elements may be confusable and sensitive to their contexts, three novel training strategies, referred to as two-channel training strategy, Maximum Separable Distance training strategy and HMM Adaptive Boosting strategy, are proposed to improve the discriminative power and robustness of an HMM classifier. Following that, approaches for recognizing words, phrases and connected-digit units in visual speech are presented with exploration of level building method and Viterbi searching algorithm. The thesis also covers the studies of 3D lip tracking and visual speech mapping between different speakers. These approaches may extend the applicability of a visual speech processing system to unfavorable conditions.