Nonconvex penalised regression and post-selection least squares under high dimensions : a local asymptotic perspective

Abstract

In the realm of high-dimensional linear regression, nonconvex penalised estimators have enjoyed increasing popularity due to their much acclaimed oracle property, which holds under assumptions weaker than those typically required for convex penalised estimators to enjoy the same property. However, validity of such oracle property of nonconvex penalisation and the accompanying inference tools is questionable in the presence of many weak signals and/or a few moderate signals, which may incur substantial biases. To address this issue, the thesis aims at developing theoretically valid, computationally feasible, procedures for estimation and inference about the relationships between strong signals and response variables. We first provide a more holistic assessment of the selection and convergence properties of nonconvex penalised estimators from a local asymptotic perspective, under a framework which accommodates existence of many weak signals or a few moderate signals and, heavy tail conditions on covariates and random errors. We then show that post-selection least squares estimation has the beneficial effect of removing the bias incurred by nonconvex penalisation of moderate signals. Post-selection least squares estimators acquire convergence properties more desirable than nonconvex penalised estimators and, in the case of multiple solutions to the nonconvex optimisation program, are ratewise more robust against the choice of selected sets. In particular, the post-selection least squares method is found to improve on nonconvex penalised estimation, especially under heavy-tailed settings. Then, trustworthy bootstrap inference procedures based on nonconvex penalised estimators and their post-selection OLS estimators are developed to estimate their distribution under this flexible framework. The revised bootstrap method draws the bootstrap samples from the post-selection OLS model, and is shown to be valid under weaker conditions. Owing to its desirable theoretical properties, the residual bootstrap method based on post-selection least squares estimators is accurate generally, and can be as effective as normal approximation, even without assuming strong conditions on signal strength. Empirical results obtained from large-scale simulation studies corroborate our theoretical findings.