Are discoveries spurious? Distributions of maximum spurious correlations and their applications

Jianqing Fan Qi-Man Shao Wen-Xin Zhou

Statistics Theory and Methods mathscidoc:1912.43367

Annals of statistics, 46, (3), 989, 2018.6
Over the last two decades, many exciting variable selection methods have been developed for finding a small group of covariates that are associated with the response from a large pool. Can the discoveries by such data mining approaches be spurious due to high dimensionality and limited sample size? Can our fundamental assumptions on exogeneity of covariates needed for such variable selection be validated with the data? To answer these questions, we need to derive the distributions of the maximum spurious correlations given certain number of predictors, namely, the distribution of the correlation of a response variable Y with the best s linear combinations of p covariates X, even when X and Y are independent. When the covariance matrix of X possesses the restricted eigenvalue property, we derive such distributions for both finite s and diverging s, using Gaussian approximation and empirical process
No keywords uploaded!
[ Download ] [ 2019-12-21 11:39:58 uploaded by Jianqing_Fan ] [ 563 downloads ] [ 0 comments ]
@inproceedings{jianqing2018are,
  title={Are discoveries spurious? Distributions of maximum spurious correlations and their applications},
  author={Jianqing Fan, Qi-Man Shao, and Wen-Xin Zhou},
  url={http://archive.ymsc.tsinghua.edu.cn/pacm_paperurl/20191221113958086690927},
  booktitle={Annals of statistics},
  volume={46},
  number={3},
  pages={989},
  year={2018},
}
Jianqing Fan, Qi-Man Shao, and Wen-Xin Zhou. Are discoveries spurious? Distributions of maximum spurious correlations and their applications. 2018. Vol. 46. In Annals of statistics. pp.989. http://archive.ymsc.tsinghua.edu.cn/pacm_paperurl/20191221113958086690927.
Please log in for comment!
 
 
Contact us: office-iccm@tsinghua.edu.cn | Copyright Reserved