On weak base hypotheses and their implications for boosting regression and classification

Wenxin Jiang*

*Corresponding author for this work

Research output: Contribution to journalArticlepeer-review

15 Scopus citations

Abstract

When studying the training error and the prediction error for boosting, it is often assumed that the hypotheses returned by the base learner are weakly accurate, or are able to beat a random guesser by a certain amount of difference. It has been an open question how much this difference can be, whether it will eventually disappear in the boosting process or be bounded by a positive amount. This question is crucial for the behavior of both the training error and the prediction error. In this paper we study this problem and show affirmatively that the amount of improvement over the random guesser will be at least a positive amount for almost all possible sample realizations and for most of the commonly used base hypotheses. This has a number of implications for the prediction error, including, for example, that boosting forever may not be good and regularization may be necessary. The problem is studied by first considering an analog of AdaBoost in regression, where we study similar properties and find that, for good performance, one cannot hope to avoid regularization by just adopting the boosting device to regression.

Original languageEnglish (US)
Pages (from-to)51-73
Number of pages23
JournalAnnals of Statistics
Volume30
Issue number1
DOIs
StatePublished - Feb 2002

Keywords

  • Angular span
  • Boosting
  • Classification
  • Error bounds
  • Least squares regression
  • Matching pursuit
  • Nearest neighbor rule
  • Overfit
  • Prediction error
  • Regularization
  • Training error
  • Weak hypotheses

ASJC Scopus subject areas

  • Statistics and Probability
  • Statistics, Probability and Uncertainty

Fingerprint Dive into the research topics of 'On weak base hypotheses and their implications for boosting regression and classification'. Together they form a unique fingerprint.

Cite this