SCAN: A Scalable Neural Networks Framework Towards Compact and Efficient Models

Linfeng Zhang Institute for Interdisciplinary Information Sciences, Tsinghua University Zhanhong Tan Institute for Interdisciplinary Information Sciences, Tsinghua University Jiebo Song Institute for Interdisciplinary Information Core Technology Jingwei Chen HiSilicon Chenglong Bao Yau Mathematical Sciences Center, Tsinghua University Kaisheng Ma Institute for Interdisciplinary Information Sciences, Tsinghua University

TBD mathscidoc:2206.43018

NeurIPS, 2019.4
Remarkable achievements have been attained by deep neural networks in various applications. However, the increasing depth and width of such models also lead to explosive growth in both storage and computation, which has restricted the deployment of deep neural networks on resource-limited edge devices. To address this problem, we propose the so-called SCAN framework for networks training and inference, which is orthogonal and complementary to existing acceleration and compression methods. The proposed SCAN firstly divides neural networks into multiple sections according to their depth and constructs shallow classifiers upon the intermediate features of different sections. Moreover, attention modules and knowledge distillation are utilized to enhance the accuracy of shallow classifiers. Based on this architecture, we further propose a threshold controlled scalable inference mechanism to approach human-like sample-specific inference. Experimental results show that SCAN can be easily equipped on various neural networks without any adjustment on hyper-parameters or neural networks architectures, yielding significant performance gain on CIFAR100 and ImageNet. Codes are be released on https://github.com/ArchipLab-LinfengZhang/pytorch-scalable-neural-networks.
No keywords uploaded!
[ Download ] [ 2022-06-16 15:26:05 uploaded by Baocl ] [ 293 downloads ] [ 0 comments ]
@inproceedings{linfeng2019scan:,
  title={SCAN: A Scalable Neural Networks Framework Towards Compact and Efficient Models},
  author={Linfeng Zhang, Zhanhong Tan, Jiebo Song, Jingwei Chen, Chenglong Bao, and Kaisheng Ma},
  url={http://archive.ymsc.tsinghua.edu.cn/pacm_paperurl/20220616152605363223377},
  booktitle={NeurIPS},
  year={2019},
}
Linfeng Zhang, Zhanhong Tan, Jiebo Song, Jingwei Chen, Chenglong Bao, and Kaisheng Ma. SCAN: A Scalable Neural Networks Framework Towards Compact and Efficient Models. 2019. In NeurIPS. http://archive.ymsc.tsinghua.edu.cn/pacm_paperurl/20220616152605363223377.
Please log in for comment!
 
 
Contact us: office-iccm@tsinghua.edu.cn | Copyright Reserved