DEEP REPRESENTATION LEARNING FOR VIDEO FOUNDATION MODELS

Abstract

In this thesis, we focus on video foundation models. Specifically, we would like to investigate approaches for learning deep representations from videos, which is one of the most important topics with respect to the video foundation models. Three challenges are identified that potentially impede the advancement of foundation models in the paradigm of video understanding. (i) Current model structures for processing videos are inefficient in extracting features from videos. (ii) The frameworks for learning video representations from unannotated data are mostly inherited from images, which fail to leverage the motions between frames, and are suboptimal for learning representations from untrimmed videos. (iii) The adaptation of pre-trained video models is limited to spatiotemporal understanding tasks, while many spatial understanding tasks could potentially benefit from incorporating the temporal context between consecutive frames.