BinaryRelax: A Relaxation Approach for Training Deep Neural Networks with Quantized Weights

Penghang Yin University of California at Los Angeles Shuai Zhang University of California at Irvine Jiancheng Lyu University of California at Los Angeles Stanley Osher University of California at Irvine Yingyong Qi University of California at Irvine Jack Xin University of California at Irvine

Information Theory mathscidoc:1802.19007

arXiv , 2018
We propose BinaryRelax, a simple two-phase algorithm, for training deep neural networks with quantized weights. The set constraint that characterizes the quantization of weights is not imposed until the late stage of training, and a sequence of pseudo quantized weights is maintained. Specifically, we relax the hard constraint into a continuous regularizer via Moreau envelope, which turns out to be the squared Euclidean distance to the set of quantized weights. The pseudo quantized weights are obtained by linearly interpolating between the float weights and their quantizations. A continuation strategy is adopted to push the weights towards the quantized state by gradually increasing the regularization parameter. In the second phase, exact quantization scheme with a small learning rate is invoked to guarantee fully quantized weights. We test BinaryRelax on the benchmark CIFAR- 10 and CIFAR-100 color image datasets to demonstrate the superiority of the relaxed quantization approach and the improved accuracy over the state-of-the-art training methods. Finally, we prove the convergence of BinaryRelax under an approximate orthogonality condition.
No keywords uploaded!
[ Download ] [ 2018-02-14 11:45:34 uploaded by jack ] [ 862 downloads ] [ 0 comments ]
@inproceedings{penghang2018binaryrelax:,
  title={BinaryRelax: A Relaxation Approach for Training Deep Neural Networks with Quantized Weights},
  author={Penghang Yin, Shuai Zhang, Jiancheng Lyu, Stanley Osher, Yingyong Qi, and Jack Xin},
  url={http://archive.ymsc.tsinghua.edu.cn/pacm_paperurl/20180214114534852206931},
  booktitle={arXiv },
  year={2018},
}
Penghang Yin, Shuai Zhang, Jiancheng Lyu, Stanley Osher, Yingyong Qi, and Jack Xin. BinaryRelax: A Relaxation Approach for Training Deep Neural Networks with Quantized Weights. 2018. In arXiv . http://archive.ymsc.tsinghua.edu.cn/pacm_paperurl/20180214114534852206931.
Please log in for comment!
 
 
Contact us: office-iccm@tsinghua.edu.cn | Copyright Reserved