Interpolation between Residual and Non-Residual Networks

Zonghan Yang Institute for Artificial Intelligence, Beijing National Research Center for Information Science and Technology, Department of Computer Science and Technology, Tsinghua University Yang Liu Institute for Artificial Intelligence, Beijing National Research Center for Information Science and Technology, Department of Computer Science and Technology, Tsinghua University Chenglong Bao Yau Mathematical Sciences Center, Tsinghua University Zuoqiang Shi Department of Mathematical Sciences, Tsinghua University

Machine Learning mathscidoc:2206.41004

ICML, 2020.8
Although ordinary differential equations (ODEs) provide insights for designing network architectures, its relationship with the non-residual convolutional neural networks (CNNs) is still unclear. In this paper, we present a novel ODE model by adding a damping term. It can be shown that the proposed model can recover both a ResNet and a CNN by adjusting an interpolation coefficient. Therefore, the damped ODE model provides a unified framework for the interpretation of residual and non-residual networks. The Lyapunov analysis reveals better stability of the proposed model, and thus yields robustness improvement of the learned networks. Experiments on a number of image classification benchmarks show that the proposed model substantially improves the accuracy of ResNet and ResNeXt over the perturbed inputs from both stochastic noise and adversarial attack methods. Moreover, the loss landscape analysis demonstrates the improved robustness of our method along the attack direction.
No keywords uploaded!
[ Download ] [ 2022-06-15 22:24:10 uploaded by Baocl ] [ 42 downloads ] [ 0 comments ]
@inproceedings{zonghan2020interpolation,
  title={Interpolation between Residual and Non-Residual Networks},
  author={Zonghan Yang, Yang Liu, Chenglong Bao, and Zuoqiang Shi},
  url={http://archive.ymsc.tsinghua.edu.cn/pacm_paperurl/20220615222411506717373},
  booktitle={ICML},
  year={2020},
}
Zonghan Yang, Yang Liu, Chenglong Bao, and Zuoqiang Shi. Interpolation between Residual and Non-Residual Networks. 2020. In ICML. http://archive.ymsc.tsinghua.edu.cn/pacm_paperurl/20220615222411506717373.
Please log in for comment!
 
 
Contact us: office-iccm@tsinghua.edu.cn | Copyright Reserved