An Empirical Comparison of Voting Classification Algorithms: Bagging, Boosting, and Variants

@article{Bauer2004AnEC,
  title={An Empirical Comparison of Voting Classification Algorithms: Bagging, Boosting, and Variants},
  author={Eric Bauer and Ron Kohavi},
  journal={Machine Learning},
  year={2004},
  volume={36},
  pages={105-139}
}
Methods for voting classification algorithms, such as Bagging and AdaBoost, have been shown to be very successful in improving the accuracy of certain classifiers for artificial and real-world datasets. We review these algorithms and describe a large empirical study comparing several variants in conjunction with a decision tree inducer (three variants) and a Naive-Bayes inducer. The purpose of the study is to improve our understanding of why and when these algorithms, which use perturbation… Expand
An Experimental Comparison of Three Methods for Constructing Ensembles of Decision Trees: Bagging, Boosting, and Randomization
TLDR
The experiments show that in situations with little or no classification noise, randomization is competitive with (and perhaps slightly superior to) bagging but not as accurate as boosting, and sometimes better than randomization. Expand
adabag: An R Package for Classification with Boosting and Bagging
TLDR
In this paper, the adabag R package is introduced and AdaBoost.M1, SAMME and bagging algorithms with classification trees as base classifiers are implemented. Expand
An experimental study on diversity for bagging and boosting with linear classifiers
TLDR
Diversity measures indicated that Boosting succeeds in inducing diversity even for stable classifiers whereas Bagging does not, confirming in a quantitative way the intuitive explanation behind the success of Boosting for linear classifiers for increasing training sizes, and the poor performance of Bagging. Expand
Randomized ensemble methods for classification trees
Abstract : We propose two methods of constructing ensembles of classifiers, One method directly injects randomness into classification tree algorithms by choosing a split randomly at each node withExpand
A Comparison of Decision Tree Ensemble Creation Techniques
TLDR
An algorithm is introduced that decides when a sufficient number of classifiers has been created for an ensemble, and is shown to result in an accurate ensemble for those methods that incorporate bagging into the construction of the ensemble. Expand
A Comparison of Decision Tree Ensemble Creation Techniques
TLDR
An algorithm is introduced that decides when a sufficient number of classifiers has been created for an ensemble, and is shown to result in an accurate ensemble for those methods that incorporate bagging into the construction of the ensemble. Expand
An empirical comparison of ensemble methods based on classification trees
TLDR
An empirical comparison of the classification error of several ensemble methods based on classification trees is performed by using 14 data sets that are publicly available and that were used by Lim, Loh and Shih in 2000. Expand
Bagging and Boosting
TLDR
Bagging and boosting are examples of ensemble learning methods from data mining that combine the predictions of many different models into an ensemble prediction, but the resulting model is not as interpretable as the constituent models because they average over a large collection of predictions. Expand
Enhanced Bagging (eBagging): A Novel Approach for Ensemble Learning
TLDR
A novel modified version of bagging, named enhanced Bagging (eBagging), which uses a new mechanism (error-based bootstrapping) when constructing training sets in order to cope with this problem of random selection in ensemble learning. Expand
Constructing ensembles of classifiers using supervised projection methods based on misclassified instances
TLDR
It is shown that supervised projection algorithms can be used for ensemble construction based on the use of supervised projections to achieve both accuracy and diversity of individual classifiers and shows better robustness in presence of noise with respect to AdaBoost. Expand
...
1
2
3
4
5
...

References

SHOWING 1-10 OF 89 REFERENCES
Experiments with a New Boosting Algorithm
TLDR
This paper describes experiments carried out to assess how well AdaBoost with and without pseudo-loss, performs on real learning problems and compared boosting to Breiman's "bagging" method when used to aggregate various classifiers. Expand
Arcing Classifiers
Recent work has shown that combining multiple versions of unstable classifiers such as trees or neural nets results in reduced test set error. One of the more effective is bagging (Breiman [1996a])Expand
Boosting the margin: A new explanation for the effectiveness of voting methods
TLDR
It is shown that techniques used in the analysis of Vapnik's support vector classifiers and of neural networks with small weights can be applied to voting methods to relate the margin distribution to the test error. Expand
Bagging, Boosting, and C4.5
TLDR
Results of applying Breiman's bagging and Freund and Schapire's boosting to a system that learns decision trees and testing on a representative collection of datasets show boosting shows the greater benefit. Expand
Boosting and Naive Bayesian learning
TLDR
It is shown that boosting applied to naive Bayesian classifiers yields combination classifiers that are representationally equivalent to standard feedforward multilayer perceptrons, which are highly plausible computationally as models of animal learning. Expand
A Study of Cross-Validation and Bootstrap for Accuracy Estimation and Model Selection
TLDR
The results indicate that for real-word datasets similar to the authors', the best method to use for model selection is ten fold stratified cross validation even if computation power allows using more folds. Expand
On Bias, Variance, 0/1—Loss, and the Curse-of-Dimensionality
  • J. Friedman
  • Mathematics, Computer Science
  • Data Mining and Knowledge Discovery
  • 2004
TLDR
This work candramatically mitigate the effect of the bias associated with some simpleestimators like “naive” Bayes, and the bias induced by the curse-of-dimensionality on nearest-neighbor procedures. Expand
Boosting Decision Trees
TLDR
A constructive, incremental learning system for regression problems that models data by means of locally linear experts that does not compete for data during learning and derives asymptotic results for this method. Expand
A decision-theoretic generalization of on-line learning and an application to boosting
TLDR
The model studied can be interpreted as a broad, abstract extension of the well-studied on-line prediction model to a general decision-theoretic setting, and the multiplicative weightupdate Littlestone Warmuth rule can be adapted to this model, yielding bounds that are slightly weaker in some cases, but applicable to a considerably more general class of learning problems. Expand
Arcing the edge
Recent work has shown that adaptively reweighting the training set, growing a classifier using the new weights, and combining the classifiers constructed to date can significantly decreaseExpand
...
1
2
3
4
5
...