TOWARDS CONCISE REPRESENTATION LEARNING ON DEEP NEURAL NETWORKS

Abstract

The success of deep neural networks is greatly attributed to their strong representation learning ability from raw data. The data representations of deep neural networks contain crucial information to understand the data. For example, the image representations from Convolutional Neural Networks (CNNs) can express the low-level and high-level features from the images; the video representations from deep neural networks convey both spatial and temporal information. In this thesis, we focus on learning ’concise representations’ on deep neural networks for various high-level vision applications, including image classifica- tion, image hashing, video summarization/hashing, where the ‘concise representations’ in deep neural network is condensed or compressed representation that can be learned with high efficiency, which includes three types: 1) lightweight representations in model size, such as the representations from small neural networks; 2) condensed representations in length or width, such as binary representations in bit level and summarized representation in batch level; 3) data-efficient representations in data-size level. However, the learned data representations from current deep neural networks are further from concise, and it is generally computationally expensive to learn data representation by deep neural networks. Learning better concise representations can result in more efficient training and inference on neural networks, smaller model size, and small data size on various vision applications. To achieve it, we systematically proposed a series of techniques to learning concise representations on deep neural networks, including learning: 1) compressed repre- sentations from lightweight models by representation distillation (Chapter 2); 2) highly condensed representations such as summarized and binary representations for image and video (Chapter 3); 3) data-efficient learnable representations for Transformer architecture (Chapter 4).