Improved Boosting Algorithms Using Confidence-rated Predictions

@article{Schapire1998ImprovedBA,
  title={Improved Boosting Algorithms Using Confidence-rated Predictions},
  author={Robert E. Schapire and Yoram Singer},
  journal={Machine Learning},
  year={1998},
  volume={37},
  pages={297-336}
}
We describe several improvements to Freund and Schapire's AdaBoost boosting algorithm, particularly in a setting in which hypotheses may assign confidences to each of their predictions. We give a simplified analysis of AdaBoost in this setting, and we show how this analysis can be used to find improved parameter settings as well as a refined criterion for training weak hypotheses. We give a specific method for assigning confidences to the predictions of decision trees, a method closely related… Expand
Stopping Criterion for Boosting-Based Data Reduction Techniques: from Binary to Multiclass Problem
TLDR
The aim of the present paper is to relax the class constraint, and extend the contribution to multiclass problems, showing the benefits that the boosting-derived weighting rule brings to weighted nearest neighbor classifiers. Expand
On the Boosting Pruning Problem (short Submission)
Boosting is a powerful method for improving the predictive accuracy of classiiers. The AdaBoost algorithm of Freund and Schapire has been successfully applied to many domains 2, 10, 12] and theExpand
On the Boosting Pruning Problem
TLDR
It is pointed out that the boosting pruning problem is intractable even to approximate and a margin-based theoretical heuristic is suggested for this problem. Expand
MP-Boost: A Multiple-Pivot Boosting Algorithm and Its Application to Text Categorization
TLDR
An improved version of AdaBoost.MH, called MP-Boost, obtained by selecting, at each iteration of the boosting process, not one but several pivot terms, one for each category, so that each iteration generates the best possible decision stump. Expand
Special Invited Paper-Additive logistic regression: A statistical view of boosting
Boosting is one of the most important recent developments in classification methodology. Boosting works by sequentially applying a classification algorithm to reweighted versions of the training dataExpand
Adatree 2 : Boosting to build decision trees or Improving Adatree with soft splitting rules
We extend the framework of Adaboost so that it builds a smooth ed decision tree rather than a neural network. The proposed method, “Adatree 2”, is derived from the assumption of a probabilisticExpand
Boosting with Bayesian stumps
  • D. Denison
  • Mathematics, Computer Science
  • Stat. Comput.
  • 2001
TLDR
This paper reviews some of the commonly used methods for performing boosting and shows how they can be fit into a Bayesian setup at each iteration of the algorithm. Expand
Additive Logistic Regression : a Statistical
TLDR
This work develops more direct approximations of boosting that exhibit performance comparable to other recently proposed multi-class generalizations of boosting, and suggests a minor modiication to boosting that can reduce computation, often by factors of 10 to 50. Expand
A Multiclass Extension To The Brownboost Algorithm
TLDR
This paper proposes a natural multiclass extension to the basic algorithm, incorporating error-correcting output codes and a multiclass gain measure, and test two-class and multiclass versions of the algorithm on a number of real and simulated data sets with artificial class noise, showing that Brownboost consistently outperforms Adaboost in these situations. Expand
The Synergy Between PAV and AdaBoost
TLDR
It is shown how PAV may be applied to a weak hypothesis to yield a new weak hypothesis which is in a sense an ideal confidence rated prediction and that this leads to an optimal updating for AdaBoost. Expand
...
1
2
3
4
5
...

References

SHOWING 1-10 OF 54 REFERENCES
Experiments with a New Boosting Algorithm
TLDR
This paper describes experiments carried out to assess how well AdaBoost with and without pseudo-loss, performs on real learning problems and compared boosting to Breiman's "bagging" method when used to aggregate various classifiers. Expand
Using output codes to boost multiclass learning problems
TLDR
This paper describes a new technique for multiclass learning problems by combining Freund and Schapire's boosting algorithm with the main ideas of Diet- terich and Bakiri's method of error-correcting output codes (ECOC), and shows that the new hybrid method has advantages of both. Expand
Special Invited Paper-Additive logistic regression: A statistical view of boosting
Boosting is one of the most important recent developments in classification methodology. Boosting works by sequentially applying a classification algorithm to reweighted versions of the training dataExpand
A decision-theoretic generalization of on-line learning and an application to boosting
TLDR
The model studied can be interpreted as a broad, abstract extension of the well-studied on-line prediction model to a general decision-theoretic setting, and the multiplicative weightupdate Littlestone Warmuth rule can be adapted to this model, yielding bounds that are slightly weaker in some cases, but applicable to a considerably more general class of learning problems. Expand
Boosting Decision Trees
TLDR
A constructive, incremental learning system for regression problems that models data by means of locally linear experts that does not compete for data during learning and derives asymptotic results for this method. Expand
An Efficient Boosting Algorithm for Combining Preferences
TLDR
This work describes and analyze an efficient algorithm called RankBoost for combining preferences based on the boosting approach to machine learning, and gives theoretical results describing the algorithm's behavior both on the training data, and on new test data not seen during training. Expand
An Empirical Evaluation of Bagging and Boosting
TLDR
The results clearly show that even though Bagging almost always produces a better classifier than any of its individual component classifiers and is relatively impervious to overfitting, it does not generalize any better than a baseline neural-network ensemble method. Expand
Using and combining predictors that specialize
TLDR
It is shown how to transform algorithms that assume that all experts are always awake to algorithms that do not require this assumption, and how to derive corresponding loss bounds. Expand
Arcing Classifiers
Recent work has shown that combining multiple versions of unstable classifiers such as trees or neural nets results in reduced test set error. One of the more effective is bagging (Breiman [1996a])Expand
On the Boosting Ability of Top-Down Decision Tree Learning Algorithms
TLDR
This work analyzes the performance of top?down algorithms for decision tree learning and proves that some popular and empirically successful heuristics that are base on first principles meet the criteria of an independently motivated theoretical model. Expand
...
1
2
3
4
5
...