No abstract uploaded!

No abstract uploaded!

Processing streaming data as they arrive is often necessary for high dimensional data analysis. In this paper, we analyze the convergence of a subspace online PCA iteration, as a followup of the recent work of Li, Wang, Liu, and Zhang [Math. Program., Ser. B, DOI 10.1007/s10107-017-1182-z] who considered the case for the most significant principal component only, i.e., a single vector. Under the sub-Gaussian assumption, we obtain a finite-sample error bound that closely matches the minimax information lower bound of Vu and Lei [Ann. Statist. 41:6 (2013), 2905-2947].

Frequentists' inference often delivers point estimators associated with condence intervals or sets for parameters of interest. Constructing the condence intervals or sets requires understanding the sampling distributions of the point estimators, which, in many but not all cases, are related to asymptotic Normal distributions ensured by central limit theorems. Although previous literature has established various forms of central limit theorems for statistical inference in super population models, we still need general and convenient forms of central limit theorems for some randomization-based causal analysis of experimental data, where the parameters of interests are functions of a nite population and randomness comes solely from the treatment assignment. We use central limit theorems for sample surveys and rank statistics to establish general forms of the nite population central limit theorems that are particularly useful for proving asymptotic distributions of randomization tests under the sharp null hypothesis of zero individual causal eects, and for obtaining the asymptotic repeated sampling distributions of the causal eect estimators. The new central limit theorems hold for general experimental designs with multiple treatment levels, multiple treatment factors and vector outcomes, and are immediately applicable for studying the asymptotic properties of many methods in causal inference, including instrumental variable, regression adjustment, rerandomization, clustered randomized experiments, and so on. Previously, the asymptotic properties of these problems are often based on heuristic arguments, which in fact rely on general forms of nite population central limit theorems that have not been established before. Our new theorems ll in this gap by providing more solid theoretical foundation for asymptotic randomization-based causal inference.

The Mumford–Shah model is one of the most important image segmentation models and has been studied extensively in the last twenty years. In this paper, we propose a two-stage segmentation method based on the Mumford–Shah model. The first stage of our method is to find a smooth solution g to a convex variant of the Mumford–Shah model. Once g is obtained, then in the second stage the segmentation is done by thresholding g into different phases. The thresholds can be given by the users or can be obtained automatically using any clustering methods. Because of the convexity of the model, g can be solved efficiently by techniques like the split-Bregman algorithm or the Chambolle–Pock method. We prove that our method is convergent and that the solution g is always unique. In our method, there is no need to specify the number of segments K (K ≥ 2) before finding g. We can obtain any K-phase segmentations by choosing (K − 1) thresholds after g is found in the first stage, and in the second stage there is no need to recompute g if the thresholds are changed to reveal different segmentation features in the image. Experimental results show that our two-stage method performs better than many standard two-phase or multiphase segmentation methods for very general images, including antimass, tubular, MRI, noisy, and blurry images.

In this paper, we analyze the minimization of seminorms ∥L · ∥ on R^n under the constraint of a bounded I-divergence D(b,H·) for rather general linear operators H and L. The I-divergence is also known as Kullback–Leibler divergence and appears in many models in imaging science, in particular when dealing with Poisson data but also in the case of multiplicative Gamma noise. Often H represents, e.g., a linear blur operator and L is some discrete derivative or frame analysis operator. A central part of this paper consists in proving relations between the parameters of I-divergence constrained and penalized problems. To solve the I-divergence constrained problem, we consider various first-order primal–dual algorithms which reduce the problem to the solution of certain proximal minimization problems in each iteration step. One of these proximation problems is an I-divergence constrained least-squares problem which can be solved based on Morozov’s discrepancy principle by a Newton method. We prove that these algorithms produce not only a sequence of vectors which converges to a minimizer of the constrained problem but also a sequence of parameters which converges to a regularization parameter so that the corresponding penalized problem has the same solution. Furthermore, we derive a rule for automatically setting the constraint parameter for data corrupted by multiplicative Gamma noise. The performance of the various algorithms is finally demonstrated for different image restoration tasks both for images corrupted by Poisson noise and multiplicative Gamma noise.

The total variation (TV) model is attractive in that it is able to preserve sharp attributes in images. However, the restored images from TV-based methods do not usually stay in a given dynamic range, and hence projection is required to bring them back into the dynamic range for visual presentation or for storage in digital media. This will affect the accuracy of the restoration as the projected image will no longer be the minimizer of the given TV model. In this paper, we show that one can get much more accurate solutions by imposing box constraints on the TV models and solving the resulting constrained models. Our numerical results show that for some images where there are many pixels with values lying on the boundary of the dynamic range, the gain can be as great as 10.28 decibel in the peak signal-to-noise ratio. One traditional hindrance using the constrained model is that it is difficult to solve. However, in this paper, we propose using the alternating direction method of multipliers (ADMM) to solve the constrained models. This leads to a fast and convergent algorithm that is applicable for both Gaussian and impulse noise. Numerical results show that our ADMM algorithm is better than some state-of-the-art algorithms for unconstrained models in terms of both accuracy and robustness with respect to the regularization parameter.

The alternating direction method of multipliers (ADMM) is a popular and efficient first-order method that has recently found numerous applications, and the proximal ADMM is an important variant of it. The main contributions of this paper are the proposition and the analysis of a class of inertial proximal ADMMs, which unify the basic ideas of the inertial proximal point method and the proximal ADMM, for linearly constrained separable convex optimization. This class of methods are of inertial nature because at each iteration the proximal ADMM is applied to a point extrapolated at the current iterate in the direction of last movement. The recently proposed inertial primal-dual algorithm [A. Chambolle and T. Pock, On the ergodic convergence rates of a first-order primaldual algorithm, preprint, 2014, Algorithm 3] and the inertial linearized ADMM [C. Chen, S. Ma, and J. Yang, arXiv:1407.8238, eq. (3.23)] are covered as special cases. The proposed algorithmic framework is very general in the sense that the weighting matrices in the proximal terms are allowed to be only positive semidefinite, but not necessarily positive definite as required by existing methods of the same kind. By setting the two proximal terms to zero, we obtain an inertial variant of the classical ADMM, which is to the best of our knowledge new. We carry out a unified analysis for the entire class of methods under very mild assumptions. In particular, convergence, as well as asymptotic $o(1/\sqrt{k})$ and nonasymptotic $O(1/\sqrt{k})$ rates of convergence, are established for the best primal function value and feasibility residues, where k denotes the iteration counter. The global iterate convergence of the generated sequence is established under an additional assumption. We also present extensive experimental results on total variation–based image reconstruction problems to illustrate the profits gained by introducing the inertial extrapolation steps.

We propose a variational approach to obtain superresolution images from multiple low-resolution frames extracted from video clips. First the displacement between the lowresolution frames and the reference frame are computed by an optical flow algorithm. Then a low-rank model is used to construct the reference frame in high-resolution by incorporating the information of the low-resolution frames. The model has two terms: a 2-norm data fidelity term and a nuclear-norm regularization term. Alternating direction method of multipliers is used to solve the model. Comparison of our methods with other models on synthetic and real video clips show that our resulting images are more accurate with less artifacts. It also provides much finer and discernable details.

No abstract uploaded!

This paper focuses on coordinate update methods, which are useful for solving problems involving large or high-dimensional datasets. They decompose a problem into simple subproblems, where each updates one, or a small block of, variables while fixing others. These methods can deal with linear and nonlinear mappings, smooth and nonsmooth functions, as well as convex and nonconvex problems. In addition, they are easy to parallelize. The great performance of coordinate update methods depends on solving simple sub-problems. To derive simple subproblems for several new classes of applications, this paper systematically studies coordinate-friendly operators that perform low-cost coordinate updates. Based on the discovered coordinate friendly operators, as well as operator splitting techniques, we obtain new coordinate update algorithms for a variety of problems in machine learning, image processing, as well as sub-areas of optimization. Several problems are treated with coordinate update for the first time in history. The obtained algorithms are scalable to large instances through parallel and even asynchronous computing. We present numerical examples to illustrate how effective these algorithms are.

Finding a fixed point to a nonexpansive operator, i.e., x = Tx, abstracts many problems in numerical linear algebra, optimization, and other areas of data sciences. To solve xed- point problems, we propose ARock, an algorithmic framework in which multiple agents (machines, processors, or cores) update x in an asynchronous parallel fashion. Asynchrony is crucial to parallel computing since it reduces synchronization wait, relaxes communication bottleneck, and thus speeds up computing significantly. At each step of ARock, an agent updates a randomly selected coordinate xi based on possibly out-of-date information on x. The agents share x through either global memory or communication. If writing xi is atomic, the agents can read and write x without memory locks. We prove that if the nonexpansive operator T has a fixed point, then with probability one, ARock generates a sequence that converges to a fixed point of T. Our conditions on T and step sizes are weaker than comparable work. Linear convergence is obtained under suitable assumptions. We propose special cases of ARock for linear systems, convex optimization, machine learning, as well as distributed and decentralized consensus problems. Numerical experiments of solving sparse logistic regression problems are presented.

Motivated by big data applications, first-order methods have been extremely popular in recent years. However, naive gradient methods generally converge slowly. Hence, much efforts have been made to accelerate various first-order methods. This paper proposes two accelerated methods towards solving structured linearly constrained convex programming, for which we assume composite convex objective that is the sum of a differentiable function and a possibly nondifferentiable one. The first method is the accelerated linearized augmented Lagrangian method (LALM). At each update to the primal variable, it allows linearization to the differentiable function and also the augmented term, and thus it enables easy subproblems. Assuming merely convexity, we show that LALM owns O(1/t) convergence if parameters are kept fixed during all the iterations and can be accelerated to O(1/t^2) if the parameters are adapted, where t is the number of total iterations. The second method is the accelerated linearized alternating direction method of multipliers (LADMM). In addition to the composite convexity, it further assumes two-block structure on the objective. Different from classic ADMM, our method allows linearization to the objective and also augmented term to make the update simple. Assuming strong convexity on one block variable, we show that LADMM also enjoys O(1=t2) convergence with adaptive parameters. This result is a significant improvement over that in [Goldstein et. al, SIIMS'14], which requires strong convexity on both block variables and no linearization to the objective or augmented term. Numerical experiments are performed on quadratic programming, image denoising, and support vector machine. The proposed accelerated methods are compared to nonaccelerated ones and also existing accelerated methods. The results demonstrate the validness of acceleration and superior performance of the proposed methods over existing ones.

Nonconvex optimization arises in many areas of computational science and engineering. However, most nonconvex optimization algorithms are only known to have local convergence or subsequence convergence properties. In this paper, we propose an algorithm for nonconvex optimization and establish its global convergence (of the whole sequence) to a critical point. In addition, we give its asymptotic convergence rate and numerically demonstrate its efficiency. In our algorithm, the variables of the underlying problem are either treated as one block or multiple disjoint blocks. It is assumed that each non-differentiable component of the objective function, or each constraint, applies only to one block of variables. The differentiable components of the objective function, however, can involve multiple blocks of variables together. Our algorithm updates one block of variables at a time by minimizing a certain prox-linear surrogate, along with an extrapolation to accelerate its convergence. The order of update can be either deterministically cyclic or randomly shuffled for each cycle. In fact, our convergence analysis only needs that each block be updated at least once in every fixed number of iterations. We show its global convergence (of the whole sequence) to a critical point under fairly loose conditions including, in particular, the Kurdyka–Łojasiewicz condition, which is satisfied by a broad class of nonconvex/nonsmooth applications. These results, of course, remain valid when the underlying problem is convex.We apply our convergence results to the coordinate descent iteration for non-convex regularized linear regression, as well as amodified rank-one residue iteration for nonnegative matrix factorization. We show that both applications have global convergence. Numerically,we tested our algorithm on nonnegative matrix and tensor factorization problems, where random shuffling clearly improves the chance to avoid low-quality local solutions.

The stochastic gradient (SG) method can quickly solve a problem with a large number of components in the objective, or a stochastic optimization problem, to a moderate accuracy. The block coordinate descent/update (BCD) method, on the other hand, can quickly solve problems with multiple (blocks of) variables. This paper introduces a method that combines the great features of SG and BCD for problems with many components in the objective and with multiple (blocks of) variables. This paper proposes a block SG (BSG) method for both convex and nonconvex programs. BSG generalizes SG by updating all the blocks of variables in the Gauss–Seidel type (updating the current block depends on the previously updated block), in either a fixed or randomly shuffled order. Although BSG has slightly more work at each iteration, it typically outperforms SG because of BSG’s Gauss–Seidel updates and larger step sizes, the latter of which are determined by the smaller per-block Lipschitz constants. The convergence of BSG is established for both convex and nonconvex cases. In the convex case, BSG has the same order of convergence rate as SG. In the nonconvex case, its convergence is established in terms of the expected violation of a first-order optimality condition. In both cases our analysis is nontrivial since the typical unbiasedness assumption no longer holds. BSG is numerically evaluated on the following problems: stochastic least squares and logistic regression, which are convex, and low-rank tensor recovery and bilinear logistic regression, which are nonconvex. On the convex problems, BSG performed significantly better than SG. On the nonconvex problems, BSG significantly outperformed the deterministic BCD method because the latter tends to stagnate early near local minimizers. Overall, BSG inherits the benefits of both SG approximation and block coordinate updates and is especially useful for solving large-scale nonconvex problems.

This paper considers regularized block multiconvex optimization, where the feasible set and objective function are generally nonconvex but convex in each block of variables. It also accepts nonconvex blocks and requires these blocks to be updated by proximal minimization. We review some interesting applications and propose a generalized block coordinate descent method. Under certain conditions, we show that any limit point satisfies the Nash equilibrium conditions. Furthermore, we establish global convergence and estimate the asymptotic convergence rate of the method by assuming a property based on the Kurdyka-Lojasiewicz inequality. The proposed algorithms are tested on nonnegative matrix and tensor factorization, as well as matrix and tensor recovery from incomplete observations. The tests include synthetic data and hyperspectral data, as well as image sets from the CBCL and ORL databases. Compared to the existing state-of-the-art algorithms, the proposed algorithms demonstrate superior performance in both speed and solution quality. The MATLAB code of nonnegative matrix/tensor decomposition and completion, along with a few demos, are accessible from the authors' homepages.

Adult stem cells, which exist throughout the body, multiply by cell division to replenish dying cells or to promote regeneration to repair damaged tissues. To perform these functions during the lifetime of organs or tissues, stem cells need to maintain their populations in a faithful distribution of their epigenetic states, which are suscepti- ble to stochastic fluctuations during each cell division, unexpected injury, and potential genetic mutations that occur during many cell divisions. However, it remains unclear how the three processes of differentiation, proliferation, and apoptosis in regulating stem cells collectively manage these challenging tasks. Here, without consid- ering molecular details, we propose a genetic optimal control model for adult stem cell regeneration that includes the three fundamental processes, along with cell division and adaptation based on differ- ential fitnesses of phenotypes. In the model, stem cells with a distri- bution of epigenetic states are required to maximize expected performance after each cell division. We show that heteroge- neous proliferation that depends on the epigenetic states of stem cells can improve the maintenance of stem cell distributions to create balanced populations. A control strategy during each cell division leads to a feedback mechanism involving heterogeneous proliferation that can accelerate regeneration with less fluctuation in the stem cell population. When mutation is allowed, apoptosis evolves to maximize the performance during homeostasis after multiple cell divisions. The overall results highlight the importance of cross-talk between genetic and epigenetic regulation and the performance objectives during homeostasis in shaping a desirable heterogeneous distribution of stem cells in epigenetic states.

In this ar ti cle, the authors de vel oped an al ter na tive meth od ol ogy to cal cu late theoption- adjusted spread for the mortgage- backed se cu ri ties us ing par tial dif fer ential equa tion tech nique. The nu meri cal im ple men ta tion is dis cussed in de tail, includ ing the con ver gence and er ror analy sis. This ap proach pro vides us a fast andac cu rate way to pric ing MBS.

Mean-Field Games are games with a continuum of players that incorporate the time-dimension through a control-theoretic approach. Recently, simpler approaches relying on the Best Reply Strategy have been proposed. They assume that the agents navigate their strategies towards their goal by taking the direction of steepest descent of their cost function (i.e. the opposite of the utility function). In this paper, we explore the link between Mean-Field Games and the Best Reply Strategy approach. This is done by introducing a Model Predictive Control framework, which consists of setting the Mean-Field Game over a short time interval which recedes as time moves on. We show that the Model Predictive Control offers a compromise between a possibly unrealistic Mean-Field Game approach and the sub-optimal Best Reply Strategy.

This paper is concerned with a mean-reversion trading rule. In contrast to most market models treated in the literature, the underlying market is solely determined by a two-state Markov chain. The major advantage of such Markov chain model is its striking simplicity and yet its capability of capturing various market movements. The purpose of this paper is to study an optimal trading rule under such a model. The objective of the problem under consideration is to nd a sequence stopping (buying and selling) times so as to maximize an expected return. Under some suitable conditions, explicit solutions to the associated HJ equations (variational inequalities) are obtained. The optimal stopping times are given in terms of a set of threshold levels. A verication theorem is provided to justify their optimality. Finally, a numerical example is provided to illustrate the results.

This paper is concerned with an optimal strategy for simultaneously trading of a pair of stocks. The idea of pairs trading is to monitor their price movements and compare their relative strength over time. A pairs trade is triggered by their prices divergence and consists of a pair of positions to short the strong stock and to long the weak one. Such a strategy bets on the reversal of their price strengths. From the viewpoint of technical tractability, typical pairs-trading models usually assume a difference of the stock prices satisfies a mean-reversion equation. In this paper, we consider the optimal pairs-trading problem by allowing the stock prices to follow general geometric Brownian motions. The objective is to trade the pairs over time to maximize an overall return with a fixed commission cost for each transaction. The optimal policy is characterized by threshold curves obtained by solving the associated HJB equations. Numerical examples are included to demonstrate the dependence of our trading rules on various parameters and to illustrate how to implement the results in practice.

It is well known that the nonlinear filtering problem has important applications in both military and civil industries. The central problem of nonlinear filtering is to solve the Duncan-Mortensen-Zakai (DMZ) equation in real time and in a memoryless manner. In this paper, we shall extend the algorithm developed previously by S.-T. Yau and the second author to the most general setting of nonlinear filterings, where the explicit time-dependence is in the drift term, observation term, and the variance of the noises could be a matrix of functions of both time and the states. To preserve the off-line virture of the algorithm, necessary modifications are illustrated clearly. Moreover, it is shown rigorously that the approximated solution obtained by the algorithm converges to the real solution in the $L^1$ sense. And the precise error has been estimated. Finally, the numerical simulation support the feasibility and efficiency of our algorithm.

In this paper, we investigate the Hermite spectral method (HSM) to numerically solve the forward Kolmogorov equation (FKE). A useful guideline of choosing the scaling factor of the generalized Hermite functions is given in this paper. It greatly improves the resolution of HSM. The convergence rate of HSM to FKE is analyzed in the suitable function space and has been verified by the numerical simulation. As an important application and our primary motivation to study the HSM to FKE, we work on the implementation of the nonlinear filtering (NLF) problems with a real-time algorithm developed by S.-T. Yau and the second author in 2008. The HSM to FKE is served as the off-line computation in this algorithm. The translating factor of the generalized Hermite functions and the moving-window technique are introduced to deal with the drifting of the posterior conditional density function of the states in the on-line experiments. Two numerical experiments of NLF problems are carried out to illustrate the feasibility of our algorithm. Moreover, our algorithm surpasses the particle filters as a real-time solver to NLF.

The surveys in the field of nonlinear filtering (NLF) are enumerous. Most of them are application-oriented and served as the tutorials for the practioners. The local approaches, including Kalman filter and its invariants, have already been studied from various point of views, due to its off-the-shelf implementation and wide applications. However, it cannot give good estimation of the states in highly nonlinear system or with non-Gaussian initial conditional density functions. Moreover, while the local methods only approximate the mean and variance, the global ones seek the way to directly obtain the conditional density function of the states. Consequently, all the statistical information is acquired. In this survey, we shall briefly go through the local approaches and put emphases on the existing three major global approaches: finite-dimensional NLF, sequential Monte Carlo methods (particle filter) and the {Yau-Yau's} on- and off-line solver of Duncan-Mortensen-Zakai's equation {\cite{YY_08}}. The discussions are mainly from the mathematical point of view.

No abstract uploaded!