Please use this identifier to cite or link to this item: https://doi.org/10.1109/ACCESS.2019.2957203
Title: Pruning Blocks for CNN Compression and Acceleration via Online Ensemble Distillation
Authors: Wang, Z.
Lin, S. 
Xie, J.
Lin, Y.
Keywords: Fast iterative shrinkage-thresholding algorithm
model compression and acceleration
network pruning
online ensemble distillation
Issue Date: 2019
Publisher: Institute of Electrical and Electronics Engineers Inc.
Citation: Wang, Z., Lin, S., Xie, J., Lin, Y. (2019). Pruning Blocks for CNN Compression and Acceleration via Online Ensemble Distillation. IEEE Access 7 : 175703-175716. ScholarBank@NUS Repository. https://doi.org/10.1109/ACCESS.2019.2957203
Rights: Attribution-NonCommercial-NoDerivatives 4.0 International
Abstract: In this paper, we propose an online ensemble distillation (OED) method to automatically prune blocks/layers of a target network by transferring the knowledge from a strong teacher in an end-to-end manner. To accomplish this, we first introduce a soft mask to scale the output of each block in the target network and enforce the sparsity of the mask by sparsity regularization. Then, a strong teacher network is constructed online by replicating the same target networks and ensembling the discriminative features from each target as its new features. Cooperative learning between multiple target networks and the teacher network is further conducted in a closed-loop form, which improves their performance. To solve the optimization problem in an end-to-end manner, we employ the fast iterative shrinkage-thresholding algorithm to fast and reliably remove the redundant blocks, in which the corresponding soft masks are equal to zero. Compared to other structured pruning methods with iterative fine-tuning, the proposed OED is trained more efficiently in one training cycle. Extensive experiments demonstrate the effectiveness of OED, which can not only simultaneously compress and accelerate a variety of CNN architectures but also enhance the robustness of the pruned networks. © 2013 IEEE.
Source Title: IEEE Access
URI: https://scholarbank.nus.edu.sg/handle/10635/213735
ISSN: 21693536
DOI: 10.1109/ACCESS.2019.2957203
Rights: Attribution-NonCommercial-NoDerivatives 4.0 International
Appears in Collections:Staff Publications
Elements

Show full item record
Files in This Item:
File Description SizeFormatAccess SettingsVersion 
10_1109_ACCESS_2019_2957203.pdf7.51 MBAdobe PDF

OPEN

NoneView/Download

SCOPUSTM   
Citations

17
checked on Jan 26, 2023

Page view(s)

72
checked on Jan 26, 2023

Google ScholarTM

Check

Altmetric


This item is licensed under a Creative Commons License Creative Commons