Audio-visual interaction in multimedia

Abstract

The bimodality of speech makes it clear that audio-visual interaction is a very important issue in personal communication applications. The research and development of multimedia communication systems should account for this interaction. In this paper a number of areas related to audio-visual interaction, such as automatic lipreading, speech-driven talking heads, and lip synchronization, are addressed. In particular, a new trend in video coding research namely joint audio-video coding is discussed. Given that mouth movements are very difficult to code because of its rapid, complex, and non-rigid motion, it is explained how having extra help from the acoustic signal can enable the coding of mouth movements more efficiently.