Self-Distillation: Towards Efficient and Compact Neural Networks

Linfeng Zhang Institute for Interdisciplinary Information Sciences, Tsinghua University, 12442 Beijing, Beijing, China Chenglong Bao Mathematics, National University of Singapore, 37580 Singapore, Singapore, Singapore Kaisheng Ma cse, Pennsylvania State University University Park, 311285 University Park, Pennsylvania, United States, 16802

TBD mathscidoc:2206.43004

IEEE Transactions on Pattern Analysis and Machine Intelligence, 14, (8), 1-1, 2021.3
Remarkable achievements have been obtained by deep neural networks in the last several years. However, the breakthrough in neural networks accuracy is always accompanied by explosive growth of computation and parameters, which leads to a severe limitation of model deployment. In this paper, we propose a novel knowledge distillation technique named self-distillation to address this problem. Self-distillation attaches several attention modules and shallow classifiers at different depths of neural networks and distills knowledge from the deepest classifier to the shallower classifiers. Different from the conventional knowledge distillation methods where the knowledge of the teacher model is transferred to another student model, self-distillation can be considered as knowledge transfer in the same model - from the deeper layers to the shallow layers. Moreover, the additional classifiers in self-distillation allow the neural network to work in a dynamic manner, which leads to a much higher acceleration. Experiments demonstrate that self-distillation has consistent and significant effectiveness on various neural networks and datasets. On average, 3.49% and 2.32% accuracy boost are observed on CIFAR100 and ImageNet. Besides, experiments show that self-distillation can be combined with other model compression methods, including knowledge distillation, pruning and lightweight model design.
No keywords uploaded!
[ Download ] [ 2022-06-13 22:07:29 uploaded by Baocl ] [ 303 downloads ] [ 0 comments ]
@inproceedings{linfeng2021self-distillation:,
  title={Self-Distillation: Towards Efficient and Compact Neural Networks},
  author={Linfeng Zhang, Chenglong Bao, and Kaisheng Ma},
  url={http://archive.ymsc.tsinghua.edu.cn/pacm_paperurl/20220613220729815636353},
  booktitle={IEEE Transactions on Pattern Analysis and Machine Intelligence},
  volume={14},
  number={8},
  pages={1-1},
  year={2021},
}
Linfeng Zhang, Chenglong Bao, and Kaisheng Ma. Self-Distillation: Towards Efficient and Compact Neural Networks. 2021. Vol. 14. In IEEE Transactions on Pattern Analysis and Machine Intelligence. pp.1-1. http://archive.ymsc.tsinghua.edu.cn/pacm_paperurl/20220613220729815636353.
Please log in for comment!
 
 
Contact us: office-iccm@tsinghua.edu.cn | Copyright Reserved