New frame interpolation scheme for talking head sequences

Abstract

A video codec typically skips frames to satisfy the bitrate constraint. This results in jerky motion and loss of lip synchronization in sequences of talking persons. In this paper, we propose a frame interpolation technique to solve these problems. Main components of this technique include: foreground/background segmentation, mesh-based frame interpolation, and image analysis/synthesis for mouth movements. This technique generates smooth head motion and renders lip motion that is synchronized with the voice of the person.