Stochastic gradient (SG) method has been popularly applied to solve optimization problems with objective that is stochastic or an average of many functions. Most existing works on SG assume that the underlying problem is unconstrained or has an easy-to-project constraint set. In this paper, we consider problems that have a stochastic objective and also many functional constraints. For such problems, it could be extremely expensive to project a point to the feasible set, or even compute subgradient and/or function value of all constraint functions. To find solutions of these problems, we propose a novel SG method based on the augmented Lagrangian function. Within every iteration, it inquires a stochastic subgradient of the objective, a subgradient and function value of one randomly sampled constraint function, and function value of another sampled constraint function. Hence, the per-iteration complexity is low. We establish its convergence rate for convex and also strongly convex problems. It can achieve the optimal O (1/\sqrt {k}) convergence rate for convex case and nearly optimal O (1/\sqrt {k}) rate for strongly convex case. Numerical experiments on quadratically constrained quadratic programming are conducted to demonstrate its efficiency.

Recent several years have witnessed the surge of asynchronous (async-) parallel computing methods due to the extremely big data involved in many modern applications and also the advancement of multi-core machines and computer clusters. In optimization, most works about async-parallel methods are on unconstrained problems or those with block separable constraints. In this paper, we propose an async-parallel method based on block coordinate update (BCU) for solving convex problems with <i>nonseparable</i> linear constraint. Running on a single node, the method becomes a novel randomized primaldual BCU for multi-block affinely constrained problems. For these problems, GaussSeidel cyclic primaldual BCU is not guaranteed to converge to an optimal solution if no additional assumptions, such as strong convexity, are made. On the contrary, assuming convexity and existence of a primaldual

Recent years have witnessed the rapid development of block coordinate update (BCU) methods, which are particularly suitable for problems involving large-sized data and/or variables. In optimization, BCU first appears as the coordinate descent method that works well for smooth problems or those with separable nonsmooth terms and/or separable constraints. As nonseparable constraints exist, BCU can be applied under primal-dual settings. In the literature, it has been shown that for weakly convex problems with nonseparable linear constraints, BCU with fully Gauss--Seidel updating rule may fail to converge and that with fully Jacobian rule can converge sublinearly. However, empirically the method with Jacobian update is usually slower than that with Gauss--Seidel rule. To maintain their advantages, we propose a hybrid Jacobian and Gauss--Seidel BCU method for solving linearly constrained multiblock

Block coordinate update (BCU) methods enjoy low per-update computational complexity because every time only one or a few block variables would need to be updated among possibly a large number of blocks. They are also easily parallelized and thus have been particularly popular for solving problems involving large-scale dataset and/or variables. In this paper, we propose a primaldual BCU method for solving linearly constrained convex program with multi-block variables. The method is an accelerated version of a primaldual algorithm proposed by the authors, which applies randomization in selecting block variables to update and establishes an <i>O</i>(1/<i>t</i>) convergence rate under convexity assumption. We show that the rate can be accelerated to O ( 1 / t 2 ) if the objective is strongly convex. In addition, if one block variable is independent of the others in the objective, we then show that the algorithm can

In this paper, we propose a randomized primaldual proximal block coordinate updating framework for a general multi-block convex optimization model with coupled objective function and linear constraints. Assuming mere convexity, we establish its <i>O</i>(1/<i>t</i>) convergence rate in terms of the objective value and feasibility measure. The framework includes several existing algorithms as special cases such as a primaldual method for bilinear saddle-point problems (PD-S), the proximal Jacobian alternating direction method of multipliers (Prox-JADMM) and a randomized variant of the ADMM for multi-block convex optimization. Our analysis recovers and/or strengthens the convergence properties of several existing algorithms. For example, for PD-S our result leads to the same order of convergence rate without the previously assumed boundedness condition on the constraint sets, and for Prox-JADMM the new

This paper introduces algorithms for the decentralized low-rank matrix completion problem. Assume a low-rank matrix W = [W ₁ ,W ₂ , ...,W _L ]. In a network, each agent observes some entries of W  . In order to recover the unobserved entries of W via decentralized computation, we factorize the unknown matrix W as the product of a public matrix X, common to all agents, and a private matrix Y = [Y ₁ ,Y ₂ , ...,Y _L ], where Y  is held by agent . Each agent alternatively updates Y  and its local estimate of X while communicating with its neighbors toward a consensus on the estimate. Once this consensus is (nearly) reached throughout the network, each agent recovers W  = XY  , and thus W is recovered. The communication cost is scalable to the number of agents, and W  and Y  are kept private to agent to a certain extent. The algorithm is accelerated by extrapolation and compares favorably to the centralized

This paper focuses on coordinate update methods, which are useful for solving problems involving large or high-dimensional datasets. They decompose a problem into simple subproblems, where each updates one, or a small block of, variables while fixing others. These methods can deal with linear and nonlinear mappings, smooth and nonsmooth functions, as well as convex and nonconvex problems. In addition, they are easy to parallelize.

Higher-order low-rank tensors naturally arise in many applications including hyperspectral data recovery, video inpainting, seismic data recon-struction, and so on. We propose a new model to recover a low-rank tensor by simultaneously performing low-rank matrix factorizations to the all-mode ma-tricizations of the underlying tensor. An alternating minimization algorithm is applied to solve the model, along with two adaptive rank-adjusting strategies when the exact rank is not known.

This paper introduces an algorithm for the nonnegative matrix factorization-and-completion problem, which aims to find nonnegative low-rank matrices <i>X</i> and <i>Y</i> so that the product <i>XY</i> approximates a nonnegative data matrix <i>M</i> whose elements are partially known (to a certain accuracy). This problem aggregates two existing problems: (i) nonnegative matrix factorization where all entries of <i>M</i> are given, and (ii) low-rank matrix completion where nonnegativity is not required. By taking the advantages of both nonnegativity and low-rankness, one can generally obtain superior results than those of just using one of the two properties. We propose to solve the non-convex constrained least-squares problem using an algorithm based on the classical alternating direction augmented Lagrangian method. Preliminary convergence properties of the algorithm and numerical simulation results are presented. Compared to

In this paper, we first study \ell_q minimization and its associated iterative reweighted algorithm for recovering sparse vectors. Unlike most existing work, we focus on <i>unconstrained</i> \ell_q minimization, for which we show a few advantages on noisy measurements and/or approximately sparse vectors. Inspired by the results in [Daubechies et al., <i>Comm. Pure Appl. Math.</i>, 63 (2010), pp. 1--38] for <i>constrained</i> \ell_q minimization, we start with a preliminary yet novel analysis for <i>unconstrained</i> \ell_q minimization, which includes convergence, error bound, and local convergence behavior. Then, the algorithm and analysis are extended to the recovery of low-rank matrices. The algorithms for both vector and matrix recovery have been compared to some state-of-the-art algorithms and show superior performance on recovering sparse vectors and low-rank matrices.

No abstract uploaded!

This work is devoted to the numerical computation of complex band structure $\k=\k(\omega)\in\mathbb C^3$ for positive $\omega$ of three dimensional isotropic dispersive or non-dispersive photonic crystals from the perspective of structured quadratic eigenvalue problems (QEPs). Our basic strategy is to fix two degrees of freedom in $\k\in\mathbb C^3$ and to view the remaining one as the eigenvalue of a quadratic operator pencil derived from Maxwell's equations. Then Yee's scheme is employed to discretize $\nabla\times$ and $\k\times$ operators in this quadratic operator pencil. Distinct from the others' works which either ignore or directly exploit the Hamiltonian structure of the spectrum of the resulting QEP, we reformulate this QEP into an equivalent $\top$-palindromic QEP to facilitate the use of superior structure-preserving algorithms. Ultimately we rely on the structured Arnoldi algorithm, namely the G$\top$SHIRA algorithm, to compute eigenvalues of a $\top$-skew-Hamiltonian pair which are near or in $[-2,2]$, a much narrower region than the whole positive real axis in the origin problem. Moreover, to accelerate the inner iterations of the G$\top$SHIRA algorithm, we propose the preconditioning technique, making most of the eigenmatrix, which can essentially be seen as the Kronecker product of three discrete Fourier transformation matrices, of the commutative discretized $\partial_x,\partial_y,\partial_z$ operators. The advantage of our method is discussed in detail and corroborated by several numerical results.

No abstract uploaded!

No abstract uploaded!

No abstract uploaded!

No abstract uploaded!

No abstract uploaded!

No abstract uploaded!

No abstract uploaded!

No abstract uploaded!

No abstract uploaded!

No abstract uploaded!

This article focuses on numerically studying the eigenstructure behavior of generalized eigenvalue problems (GEPs) arising in three dimensional (3D) source-free Maxwell's equations with magnetoelectric coupling effects which model 3D reciprocal chiral media. It is challenging to solve such a large-scale GEP efficiently. We combine the null-space free method with the inexact shift-invert residual Arnoldi method and MINRES linear solver to solve the GEP with a matrix dimension as large as 5,308,416. The eigenstructure is heavily determined by the chirality parameter $\gamma$. We show that all the eigenvalues are real and finite for a small chirality $\gamma$. For a critical value $\gamma = \gamma^*$, the GEP has $2 \times 2$ Jordan blocks at infinity eigenvalues. Numerical results demonstrate that when $\gamma$ increases from $\gamma^*$, the $2 \times 2$ Jordan block will first split into a complex conjugate eigenpair, then rapidly collide \tr{with} the real axis and bifurcate into \tr{positive (resonance) and negative eigenvalues with modulus} smaller than the other existing positive eigenvalues. The resonance band also exhibits an anticrossing interaction. Moreover, the electric and magnetic fields of the resonance modes are localized inside the structure, with only a slight amount of field leaking into the background (dielectric) material.

This article focuses on numerically studying the eigenstructure behavior of generalized eigenvalue problems (GEPs) arising in three dimensional (3D) source-free Maxwell's equations with magnetoelectric coupling effects which model 3D reciprocal chiral media. It is challenging to solve such a large-scale GEP efficiently. We combine the null-space free method with the inexact shift-invert residual Arnoldi method and MINRES linear solver to solve the GEP with a matrix dimension as large as 5,308,416. The eigenstructure is heavily determined by the chirality parameter $\gamma$. We show that all the eigenvalues are real and finite for a small chirality $\gamma$. For a critical value $\gamma = \gamma^*$, the GEP has $2 \times 2$ Jordan blocks at infinity eigenvalues. Numerical results demonstrate that when $\gamma$ increases from $\gamma^*$, the $2 \times 2$ Jordan block will first split into a complex conjugate eigenpair, then rapidly collide with the real axis and bifurcate into positive (resonance) and negative eigenvalues with modulus smaller than the other existing positive eigenvalues. The resonance band also exhibits an anticrossing interaction. Moreover, the electric and magnetic fields of the resonance modes are localized inside the structure, with only a slight amount of field leaking into the background (dielectric) material.

Full spectrum of a large sparse $\top$-palindromic quadratic eigenvalue problem ($\top$-PQEP) is considered arguably for the first time in this article. Such a problem is posed by calculation of surface Green's functions (SGFs) of mesoscopic transistors with a tremendous non-periodic cross-section. For this problem, general purpose eigensolvers are not efficient, nor is advisable to resort to the decimation method \etc to obtain the Wiener-Hopf factorization. After reviewing some rigorous understanding of SGF calculation from the perspective of $\top$-PQEP and nonlinear matrix equation, we present our new approach to this problem. In a nutshell, the unit disk where the spectrum of interest lies is broken down adaptively into pieces small enough that they each can be locally tackled by the generalized $\top$-skew-Hamiltonian implicitly restarted shift-and-invert Arnoldi (G$\top$SHIRA) algorithm with suitable shifts and other parameters, and the eigenvalues missed by this divide-and-conquer strategy can be recovered thanks to the accurate estimation provided by our newly developed scheme. Notably the novel non-equivalence deflation is proposed to avoid as much as possible duplication of nearby known eigenvalues when a new shift of G$\top$SHIRA is determined. We demonstrate our new approach by calculating the SGF of a realistic nanowire whose unit cell is described by a matrix of size $4000\times 4000$ at the density functional tight binding level, corresponding to a $8\times 8\; \mbox{nm}^{2}$ cross-section. We believe that quantum transport simulation of realistic nano-devices in the mesoscopic regime will greatly benefit from this work.