Joint audio-video processing for multimedia

Abstract

In this paper, we report recent development in the research of joint audio-visual processing for multimedia applications. These include: bimodality in speech production and perception, automatic lipreading, talking-head animation, and lip synchronization. We present in detail the enabling technologies for these applications. A new trend of research is to utilize audio-visual interaction in the coding of talking head video. We show that the marriage of speech analysis and image processing can create a number of new research opportunities.