Learning rate is arguably the most important hyper-parameter to tune when training a neural network. As manually setting right learning rate remains a cumbersome process, adaptive learning rate algorithms aim at automating such a process. Motivated by the success of the Barzilai–Borwein (BB) step-size method in many gradient descent methods for solving convex problems, this paper aims at investigating the potential of the BB method for training neural networks. With strong motivation from related convergence analysis, the BB method is generalized to adaptive learning rate of mini-batch gradient descent. The experiments showed that, in contrast to many existing methods, the proposed BB method is highly insensitive to initial learning rate, especially in terms of generalization performance. Also, the BB method showed its advantages on both learning speed and generalization performance over other available methods.

Finding the stationary states of a free energy functional is an important problem in phase field crystal (PFC) models. Many efforts have been devoted to designing numerical schemes with energy dissipation and mass conservation properties. However, most existing approaches are time-consuming due to the requirement of small effective step sizes. In this paper, we discretize the energy functional and propose efficient numerical algorithms for solving the constrained nonconvex minimization problem. A class of gradient-based approaches, which are the so-called adaptive accelerated Bregman proximal gradient (AA-BPG) methods, is proposed, and the convergence property is established without the global Lipschitz constant requirements. A practical Newton method is also designed to further accelerate the local convergence with convergence guarantee. One key feature of our algorithms is that the energy dissipation and mass conservation properties hold during the iteration process. Moreover, we develop a hybrid acceleration framework to accelerate the AA-BPG methods and most of the existing approaches through coupling with the practical Newton method. Extensive numerical experiments, including two three-dimensional periodic crystals in the Landau--Brazovskii (LB) model and a two-dimensional quasicrystal in the Lifshitz--Petrich (LP) model, demonstrate that our approaches have adaptive step sizes which lead to a significant acceleration over many existing methods when computing complex structures.

In this paper, we compute the stationary states of the multicomponent phase-field crystal model by formulating it as a block constrained minimization problem. The original infinite-dimensional non-convex minimization problem is approximated by a finite-dimensional constrained non-convex minimization problem after an appropriate spatial discretization. To efficiently solve the above optimization problem, we propose a so-called adaptive block Bregman proximal gradient (AB-BPG) algorithm that fully exploits the problem's block structure. The proposed method updates each order parameter alternatively, and the update order of blocks can be chosen in a deterministic or random manner. Besides, we choose the step size by developing a practical linear search approach such that the generated sequence either keeps energy dissipation or has a controllable subsequence with energy dissipation. The convergence property of the proposed method is established without the requirement of global Lipschitz continuity of the derivative of the bulk energy part by using the Bregman divergence. The numerical results on computing stationary ordered structures in binary, ternary, and quinary component coupled-mode Swift-Hohenberg models have shown a significant acceleration over many existing methods.

Remarkable achievements have been obtained by deep neural networks in the last several years. However, the breakthrough in neural networks accuracy is always accompanied by explosive growth of computation and parameters, which leads to a severe limitation of model deployment. In this paper, we propose a novel knowledge distillation technique named self-distillation to address this problem. Self-distillation attaches several attention modules and shallow classifiers at different depths of neural networks and distills knowledge from the deepest classifier to the shallower classifiers. Different from the conventional knowledge distillation methods where the knowledge of the teacher model is transferred to another student model, self-distillation can be considered as knowledge transfer in the same model - from the deeper layers to the shallow layers. Moreover, the additional classifiers in self-distillation allow the neural network to work in a dynamic manner, which leads to a much higher acceleration. Experiments demonstrate that self-distillation has consistent and significant effectiveness on various neural networks and datasets. On average, 3.49% and 2.32% accuracy boost are observed on CIFAR100 and ImageNet. Besides, experiments show that self-distillation can be combined with other model compression methods, including knowledge distillation, pruning and lightweight model design.

The Chan-Vese (CV) model is a classic region-based method in image segmentation. However, its piecewise constant assumption does not always hold for practical applications. Many improvements have been proposed but the issue is still far from well solved. In this work, we propose an unsupervised image segmentation approach that integrates the CV model with deep neural networks, which significantly improves the original CV model's segmentation accuracy. Our basic idea is to apply a deep neural network that maps the image into a latent space to alleviate the violation of the piecewise constant assumption in image space. We formulate this idea under the classic Bayesian framework by approximating the likelihood with an evidence lower bound (ELBO) term while keeping the prior term in the CV model. Thus, our model only needs the input image itself and does not require pre-training from external datasets. Moreover, we extend the idea to multi-phase case and dataset based unsupervised image segmentation. Extensive experiments validate the effectiveness of our model and show that the proposed method is noticeably better than other unsupervised segmentation approaches.