A general theory of hypothesis tests and confidence regions for sparse high dimensional models

Yang Ning, Han Liu

Research output: Contribution to journalArticlepeer-review

165 Scopus citations

Abstract

We consider the problem of uncertainty assessment for low dimensional components in high dimensional models. Specifically, we propose a novel decorrelated score function to handle the impact of high dimensional nuisance parameters. We consider both hypothesis tests and confidence regions for generic penalized M-estimators. Unlike most existing inferential methods which are tailored for individual models, our method provides a general framework for high dimensional inference and is applicable to a wide variety of applications. In particular, we apply this general framework to study five illustrative examples: linear regression, logistic regression, Poisson regression, Gaussian graphical model and additive hazards model. For hypothesis testing, we develop general theorems to characterize the limiting distributions of the decorrelated score test statistic under both null hypothesis and local alternatives. These results provide asymptotic guarantees on the type I errors and local powers. For confidence region construction, we show that the decorrelated score function can be used to construct point estimators that are asymptotically normal and semiparametrically efficient. We further generalize this framework to handle the settings of misspecified models. Thorough numerical results are provided to back up the developed theory.

Original languageEnglish (US)
Pages (from-to)158-195
Number of pages38
JournalAnnals of Statistics
Volume45
Issue number1
DOIs
StatePublished - Feb 1 2017

Keywords

  • Confidence interval
  • High dimensional inference
  • Hypothesis test
  • Model misspecification
  • Nuisance parameter
  • Score function
  • Sparsity

ASJC Scopus subject areas

  • Statistics and Probability
  • Statistics, Probability and Uncertainty

Fingerprint

Dive into the research topics of 'A general theory of hypothesis tests and confidence regions for sparse high dimensional models'. Together they form a unique fingerprint.

Cite this