Improving discrete AdaBoost for classification by randomization methods

Abstract

Adaboost, a typical boosting method for classification, performs well in classification problems. Many researchers have applied different types of randomization techniques to Adaboost for further improving the efficiency of classification. However, these methods of randomization seldom aim at the chance mechanism underlying the training data itself, especially at the response level. We propose a new modified Adaboost procedure which takes into account the chance mechanism. Three different methods are investigated for estimating the conditional probabilities of class labels given feature covariates, based on which the class labels are randomized within the training dataset. The first method, which we term quantile calibration, makes use of a reweighting scheme to find a reliable interval containing the conditional class probability. The second method applies Bootstrap Aggregating to obtain an equal weight ensemble vote for each class label. The third method exploits a well-known connection between the score function of AdaBoost and class probabilities under an additive logistic regression setup. Empirical results show that our new procedure successfully alleviates the overfitting problem, and in many cases improves the classification performance of Adaboost as well.