CROSS-MODALITY COMPLEMENTARITY FOR AUDIO-VISUAL SPEECH RECOGNITION

Abstract

Speech recognition is an indispensable tool for human-robot interaction. Inspired by human processes, the integration of audio and visual modalities enhances robustness in transcribing texts. However, corruption may occur in either modality or both, leading to a degradation in speech recognition. Therefore, utilizing audio-visual complementarity to mitigate corruption is essential, given the unique properties of these two modalities. To achieve this goal, I address three types of corruption through audio-visual complementarity. Mimicking human speech perception, the first part employs the visual modality to complement speeches corrupted by acoustic noise. The second part tackles the issue of speakers missing in the camera's field of view through a novel sound source localization. Finally, the third part aims to reconstruct occluded lips with the assistance of the audio modality.