Temporal Redundancy-Based Computation Reduction for 3D Convolutional Neural Networks

Abstract

Making sense of human actions in video sequences has become an essential task in video surveillance applications. In such applications, 3D CNNs have become a prime choice due to their excellent performance. However, the performance advantage offered by these networks comes at a higher computational and memory cost. In this paper, a novel method is introduced to enhance temporal similarity at minimal accuracy degradation in input video sequences for 3D CNNs, as demonstrated through video benchmarks focusing on human action recognition.

The proposed Temporal Similarity Tunnels (TST) method enhances temporal similarity in the feature maps of the initial and the subsequent frames, reducing computations in the convolutional layer of 3D CNNs. The proposed method achieves 46.5% (45%) computation reduction in the C3D network evaluated under the UCF101 (HMDB51) dataset, while maintaining an accuracy drop of <1%. Similarly, the computation reduction in 3D-MobileNets v1 and v2 is 48% (38%) at an accuracy drop of 1% (1.4%) for UCF101 dataset.