This article considers experimental design based on the strategy of rerandomization to increase the efficiency in experiments. Two aspects of rerandomization are addressed. First, we propose a two-stage allocation sample scheme for randomization inference to the units in experiments that guarantees that the difference-in-mean estimator is an unbiased estimator of the sample average treatment effect for any experiment, conserves the exactness of randomization inference, and halves the time consumption of the rerandomization design. Second, we propose a rank-based covariate-balance measure which can take into account the estimated relative weight of each covariate. Several strategies for estimating these weights using pre-experimental data are proposed. Using Monte Carlo simulations, the proposed strategies are compared to complete randomization and Mahalanobis-based rerandomization. An empirical example is given where the power of a mean difference test of electricity consumption of 54 households is increased by 99%, in comparison to complete randomization, using one of the proposed designs based on high frequency longitudinal electricity consumption data. Supplementary materials for this article are available online.

Recently a computational-based experimental design strategy called rerandomization has been proposed as an alternative or complement to traditional blocked designs. The idea of rerandomization is to remove, from consideration, those allocations with large imbalances in observed covariates according to a balance criterion, and then randomize within the set of acceptable allocations. Based on the Mahalanobis distance criterion for balancing the covariates, we show that asymptotic inference to the population, from which the units in the sample are randomly drawn, is possible using only the set of best, or ‘optimal’, allocations. Finally, we show that for the optimal and near optimal designs, the quite complex asymptotic sampling distribution derived by Li et al. (2018), is well approximated by a normal distribution.

Blocking is commonly used in randomized experiments to increase efficiency of estimation. A generalization of blocking removes allocations with imbalance in covariate distributions between treated and control units, and then randomizes within the remaining set of allocations with balance. This idea of rerandomization was formalized by Morgan and Rubin (Annals of Statistics, 2012, 40, 1263–1282), who suggested using Mahalanobis distance between treated and control covariate means as the criterion for removing unbalanced allocations. Kallus (Journal of the Royal Statistical Society, Series B: Statistical Methodology, 2018, 80, 85–112) proposed reducing the set of balanced allocations to the minimum. Here we discuss the implication of such an ‘optimal’ rerandomization design for inferences to the units in the sample and to the population from which the units in the sample were randomly drawn. We argue that, in general, it is a bad idea to seek the optimal design for an inference because that inference typically only reflects uncertainty from the random sampling of units, which is usually hypothetical, and not the randomization of units to treatment versus control.

In deconvolution in R^d, d≥1, with mixing density p(∈\mathscr{P}) and kernel h, the mixture density f_p(∈\mathscr{F}_p) is estimated with MDE f_{\hat{p}n}, having upper L1-error rate, an, in probability or in risk; \hat{p}_n∈\mathscr{P}. In one application, \mathscr{P} consists of L1-separable densities in R with differences changing sign at most J times and h(x−y) Totally Positive. When h is known and p is \tilde{q}-smooth, vanishing outside a compact in R^d, plug-in upper bounds are provided for the L2-error rate of \hat{p}_n and its [s]-th mixed partial derivative \hat{p}^{(s)}_n, via ∥f_{\hat{p}_n}−f_p∥_1, with rates (log a^{−1}_n)^{−N_1} and a^{N_2}_n, respectively, for h super-smooth and smooth; \tilde{q}∈R^+, [s] ≤ \tilde{q}, d≥1, N_1>0, N_2>0. For a_n ∼ (logn)^ζ⋅(n^{−δ}), the former rate is optimal for any δ>0 and the latter misses the optimal by the factor (logn)^ξ when δ=.5; ζ>0, ξ>0. N_1 and N_2 appear in optimal rates and lower error and risk bounds in the deconvolution literature.

Double-blind randomized controlled trials are traditionally seen as the gold standard for causal inferences as the difference-in-means estimator is an unbiased estimator of the average treatment effect in the experiment. The fact that this estimator is unbiased over all possible randomizations does not, however, mean that any given estimate is close to the true treatment effect. Similarly, while predetermined covariates will be balanced between treatment and control groups on average, large imbalances may be observed in a given experiment and the researcher may therefore want to condition on such covariates using linear regression. This article studies the theoretical properties of both the difference-in-means and OLS estimators conditional on observed differences in covariates. By deriving the statistical properties of the conditional estimators, we can establish guidance for how to deal with covariate imbalances.

Definitive screening designs (DSDs) are widely used for studying quantitative factors. However, DSDs constructed from different conference matrices are not equally good. We show DSDs using Paley’s conference matrices guarantee desirable performance (either optimal or near-optimal) under several important criteria.

We consider the edge statistics of large dimensional deformed rectangular matrices of the form $Y_t=Y+\sqrt{t}X,$ where $Y$ is a $p \times n$ deterministic signal matrix whose rank is comparable to $n$, $X$ is a $p\times n$ random noise matrix with centered i.i.d. entries with variance $n^{-1}$, and $t>0$ gives the noise level. This model is referred to as the interference-plus-noise matrix in the study of massive multiple-input multiple-output (MIMO) system, which belongs to the category of the so-called signal-plus-noise model. For the case $t=1$, the spectral statistics of this model have been studied to a certain extent in the literature \cite{DOZIER20071099,DOZIER2007678,VLM2012}. In this paper, we study the singular value and singular vector statistics of $Y_t$ around the right-most edge of the singular value spectrum in the harder regime $n^{-2/3}\ll t \ll 1$. This regime is harder than the $t=1$ case, because on one hand, the edge behavior of the empirical spectral distribution (ESD) of $YY^\top$ has a strong effect on the edge statistics of $Y_tY_t^\top$ since $t\ll 1$ is ``small", while on the other hand, the edge statistics of $Y_t$ is also not merely a perturbation of those of $Y$ since $t\gg n^{-2/3}$ is ``large". Under certain regularity assumptions on $Y,$ we prove the edge universality, eigenvalues rigidity and eigenvector delocalization for the matrices $Y_tY_t^\top$ and $Y_t^\top Y_t$. These results can be used to estimate and infer the massive MIMO system. To prove the main results, we analyze the edge behavior of the asymptotic ESD for $Y_tY_t^\top$, and establish some sharp local laws on the resolvent of $Y_tY_t^\top$. These results can be of independent interest, and used as useful inputs for many other problems regarding the spectral statistics of $Y_t$.

Large dimensional Gram type matrices are common objects in high-dimensional statistics and machine learning. In this paper, we study the limiting distribution of the edge eigenvalues for a general class of high-dimensional Gram type random matrices, including separable sample covariance matrices, sparse sample covariance matrices, bipartite stochastic block model and random Gram matrices with general variance profiles. Specifically, we prove that under (almost) sharp moment conditions and certain tractable regularity assumptions, the edge eigenvalues, i.e., the largest few eigenvalues of non-spiked Gram type random matrices or the extremal bulk eigenvalues of spiked Gram type random matrices, satisfy the Tracy-Widom distribution asymptotically. Our results can be used to construct adaptive, accurate and powerful statistics for high-dimensional statistical inference. In particular, we propose data-dependent statistics to infer the number of signals under general noise structure, test the one-sided sphericity of separable matrix, and test the structure of bipartite stochastic block model. Numerical simulations show strong support of our proposed statistics. The core of our proof is to establish the edge universality and Tracy-Widom distribution for a rectangular Dyson Brownian motion with regular initial data. This is a general strategy to study the edge statistics for high-dimensional Gram type random matrices without exploring the specific independence structure of the target matrices. It has potential to be applied to more general random matrices that are beyond the ones considered in this paper.

In this paper, we prove a CLT for the sample canonical correlation coefficients between two high-dimensional random vectors with finite rank correlations. More precisely, consider two random vectors $\wt{\bx}=\mathbf x + A \mathbf z $ and $\wt{\by}=\mathbf y + B \mathbf z $, where $\mathbf x \in \R^p$, $\mathbf y \in \R^q$ and $\mathbf z\in \R^r$ are independent random vectors with i.i.d.\;entries of mean zero and variance one, and $A \in \R^{p\times r}$ and $B\in \R^{q\times r}$ are two arbitrary deterministic matrices. Given $n$ samples of $\wt{\bx}$ and $\wt{\by}$, we stack them into two matrices $\cal X= X+AZ$ and $\cal Y= Y+BZ$, where $X\in \R^{p\times n}$, $Y\in \R^{q\times n}$ and $Z\in \R^{r\times n}$ are random matrices with i.i.d.\;entries of mean zero and variance one. Let $\wt\lambda_1 \ge \wt\lambda_2\ge \cdots \ge \wt\lambda_{r}$ be the largest $r$ eigenvalues of the sample canonical correlation (SCC) matrix $\cal C_{\cal X\cal Y}=(\cal X\cal X^\top)^{-1/2}\cal X\cal Y^\top (\cal Y\cal Y^\top)^{-1}\cal Y \cal X^\top (\cal X\cal X^\top)^{-1/2}$, and let $t_1\ge t_2 \ge \cdots\ge t_r$ be the squares of the population canonical correlation coefficients between $\wt{\bx}$ and $\wt{\by}$. Under certain moment assumptions, we show that there exists a threshold $t_c \in(0, 1)$ such that if $t_i>t_c$, then \smash{$\sqrt{n} (\wt\lambda_i-\theta_i)$} converges weakly to a centered normal distribution, where $\theta_i $ is a fixed outlier location determined by $t_i$. Our proof uses a self-adjoint linearization of the SCC matrix and a sharp local law on the inverse of the linearized matrix.

Consider two random vectors $\wt{\bx} \in \R^p$ and $\wt{\by} \in \mathbb R^q$ of the forms $\wt{\bx}=A \mathbf z + \bC_1^{1/2}\mathbf x $ and $\wt{\by}=B \mathbf z + \bC_2^{1/2}\mathbf y $, where $\mathbf x \in \R^p$, $\mathbf y \in \R^q$ and $\mathbf z\in \R^r$ are independent random vectors with i.i.d. entries of zero mean and unit variance, $\bC_1$ and $\bC_2$ are $p \times p$ and $q\times q$ deterministic population covariance matrices, and $A$ and $B$ are $p \times r$ and $q\times r$ deterministic factor loading matrices. With $n$ independent observations of $(\wt{\mathbf x},\wt{\mathbf y})$, we study the sample canonical correlations between $\wt{\bx} $ and $\wt{\by}$. We consider the high-dimensional setting with finite rank correlations, that is, ${p}/{n}\to c_1$ and ${q}/{n}\to c_2$ as $n\to \infty$ for some constants $c_1\in (0,1)$ and $c_2\in (0,1-c_1)$, and $r$ is a fixed integer. Let $t_1\ge t_2 \ge \cdots\ge t_r\ge 0$ be the squares of the nontrivial population canonical correlation coefficients between $\wt {\bx}$ and $\wt{\by}$, and let $\wt\lambda_1 \ge \wt\lambda_2\ge \cdots \ge \wt\lambda_{p\wedge q}\ge 0$ be the squares of the sample canonical correlation coefficients. If the entries of $\mathbf x$, $\mathbf y$ and $\mathbf z$ are i.i.d. Gaussian, then the following dichotomy has been shown in \cite{CCA} for a fixed threshold $t_c \in(0, 1)$: for any $1\le i \le r$, if $t_i < t_c$, then $\wt\lambda_i$ converges to the right-edge $\lambda_+$ of the limiting eigenvalue spectrum of the sample canonical correlation matrix, and moreover, $n^{2/3}(\wt\lambda_i-\lambda_+)$ converges weakly to the Tracy-Widom law; if $t_i>t_c$, then $\wt\lambda_i$ converges to a deterministic limit $\theta_i \in (\lambda_+, 1)$ that is determined by $c_1$, $c_2$ and $t_i$. In this paper, we prove that these results hold universally under the sharp fourth moment conditions on the entries of $\mathbf x$, $\mathbf y$ and $\mathbf z$. Moreover, we prove the results in full generality, in the sense that they also hold for near-degenerate $t_i$'s and for $t_i$'s that are close to the threshold $t_c$. Finally, we also provide almost sharp convergence rates for the sample canonical correlation coefficients under a general $a$-th moment assumption.