Video segmentation: Temporally-constrained graph-based optimization

Abstract

Video segmentation not only spatially performs intra-frame pixel grouping, but also temporally exploits the inter-frame coherence and variations of the grouping. Traditional approaches simply regard pixel motions as another prior in the MRF-MAP framework. Since pixel pre-grouping is inefficiently performed on every frame, the strong inter-frame correlation is largely underutilized. In this work, spatio-temporal grouping is accomplished by propagating and validating a preceding graph which encodes pixel labels for the previous frame, followed by spatial subgraph aggregation. Graph propagation is achieved by a global motion estimation which relates two frames temporally. All propagated labels are carefully validated by similarity measures. Trustworthy labels are preserved and erroneous ones removed, thus transforming the current frame segmentation into a highly constrained graph partitioning problem. All unlabeled subgraphs are spatially aggregated for the final grouping. Experimental results show that the proposed approach is highly efficient for the spatio-temporal segmentation and it produces encouraging results.