# Robust Boosting via Convex Optimization: Theory and Applications

@inproceedings{Rtsch2007RobustBV, title={Robust Boosting via Convex Optimization: Theory and Applications}, author={Gunnar R{\"a}tsch}, year={2007} }

In this work we consider statistical learning problems. A learning machine aims to extract information from a set of training examples such that it is able to predict the associated label on unseen examples. We consider the case where the resulting classification or regression rule is a combination of simple rules – also called base hypotheses. The so-called boosting algorithms iteratively find a weighted linear combination of base hypotheses that predict well on unseen data. We address the…

## Figures, Tables, and Topics from this paper

## 41 Citations

Boosting Algorithms for Maximizing the Soft Margin

- Computer Science, MathematicsNIPS
- 2007

This work presents a novel boosting algorithm, called SoftBoost, designed for sets of binary labeled examples that are not necessarily separable by convex combinations of base hypotheses, that produces a convex combination of hypotheses whose soft margin is within δ of its maximum.

Ensembles of Partially Trained SVMs with Multiplicative Updates

- Mathematics, Computer ScienceIJCAI
- 2007

This paper shows that the multiplicative update of SVM can be formulated as a Bregman projection problem, and the learning rate can then be adapted automatically, and it is shown that the proposed ensemble has efficient training and comparable or even better accuracy than the best-tuned soft-margin SVM.

On the Current State of Research in Explaining Ensemble Performance Using Margins

- Computer Science, MathematicsArXiv
- 2019

Several techniques are proposed and evidence suggesting that the generalization error of a voting classifier might be reduced by increasing the mean and decreasing the variance of the margins is provided, suggesting the current state of research in explaining ensemble performance holds.

Reducing the Overfitting of Adaboost by Controlling its Data Distribution Skewness

- Mathematics, Computer ScienceInt. J. Pattern Recognit. Artif. Intell.
- 2006

This paper derives two new regularized AdaBoost algorithms, referred to as AdaBoostKL and AdaBoostNorm2, respectively, from two smooth convex penalty functions, based on Kullback–Leibler divergence and l2 norm, and proves that their algorithms perform stage-wise gradient descent on a cost function, defined in the domain of their associated soft margins.

Entropy Regularized LPBoost

- Mathematics, Computer ScienceALT
- 2008

By simply adding a relative entropy regularization to the linear objective of LPBoost, this paper arrives at the Entropy Regularized LPBoost algorithm for which it is shown that this algorithm can be proved to have a logarithmic iteration bound.

A Tutorial Review of RKHS Methods in Machine Learning

- 2005

Over the last ten years, estimation and learning methods utilizing positive definite kernels have become rather popular, particularly in machine learning. Since these methods have a stronger…

Analysis of boosting algorithms using the smooth margin function

- Mathematics
- 2007

We introduce a useful tool for analyzing boosting algorithms called the "smooth margin function," a differentiable approximation of the usual margin for boosting algorithms. We present two boosting…

A Dynamic AdaBoost Algorithm With Adaptive Changes of Loss Function

- Mathematics, Computer ScienceIEEE Transactions on Systems, Man, and Cybernetics, Part C (Applications and Reviews)
- 2012

DAdaBoost can effectively prevent AdaBoost from overfitting, and Hoeffding inequality is adopted as a statistical tool to divide training samples into reliable samples and temporary unreliable samples.

Submitted to the Annals of Statistics ANALYSIS OF BOOSTING ALGORITHMS USING THE SMOOTH MARGIN FUNCTION ∗ By

- 2007

Columbia University and Princeton University We introduce a useful tool for analyzing boosting algorithms called the “smooth margin function”, a differentiable approximation of the usual margin for…

Approximate Reduction from AUC Maximization to 1-Norm Soft Margin Optimization

- Mathematics, Computer ScienceALT
- 2011

This paper shows that the hard margin case is approximately reduced to a soft margin optimization problem over p + n instances for which the resulting linear classifier is guaranteed to have a certain margin over pairs.

## References

SHOWING 1-10 OF 255 REFERENCES

Boosting in the Limit: Maximizing the Margin of Learned Ensembles

- Computer ScienceAAAI/IAAI
- 1998

The crucial question as to why boosting works so well in practice, and how to further improve upon it, remains mostly open, and it is concluded that no simple version of the minimum-margin story can be complete.

Improved Generalization Through Explicit Optimization of Margins

- Computer ScienceMachine Learning
- 2004

A theorem bounding the generalization performance of convex combinations in terms of general cost functions of the margin is proved, in contrast to previous results, which were stated in Terms of the particular cost function sgn(θ − margin).

Special Invited Paper-Additive logistic regression: A statistical view of boosting

- Mathematics
- 2000

Boosting is one of the most important recent developments in classification methodology. Boosting works by sequentially applying a classification algorithm to reweighted versions of the training data…

On weak base hypotheses and their implications for boosting regression and classification

- Mathematics
- 2002

When studying the training error and the prediction error for boosting, it is often assumed that the hypotheses returned by the base learner are weakly accurate, or are able to beat a random guesser…

Linear Programming Boosting via Column Generation

- Computer ScienceMachine Learning
- 2004

It is proved that for classification, minimizing the 1-norm soft margin error function directly optimizes a generalization error bound and is competitive in quality and computational cost to AdaBoost.

Constructing Boosting Algorithms from SVMs: An Application to One-Class Classification

- Computer ScienceIEEE Trans. Pattern Anal. Mach. Intell.
- 2002

We show via an equivalence of mathematical programs that a support vector (SV) algorithm can be translated into an equivalent boosting-like algorithm and vice versa. We exemplify this translation…

Robust Bounds on Generalization from the Margin Distribution

- Mathematics
- 1998

A number of results have bounded generalization of a classifier in terms of its margin on the training points. There has been some debate about whether the minimum margin is the best measure of the…

A decision-theoretic generalization of on-line learning and an application to boosting

- Computer ScienceEuroCOLT
- 1995

The model studied can be interpreted as a broad, abstract extension of the well-studied on-line prediction model to a general decision-theoretic setting, and the multiplicative weightupdate Littlestone Warmuth rule can be adapted to this model, yielding bounds that are slightly weaker in some cases, but applicable to a considerably more general class of learning problems.

Experiments with a New Boosting Algorithm

- Computer ScienceICML
- 1996

This paper describes experiments carried out to assess how well AdaBoost with and without pseudo-loss, performs on real learning problems and compared boosting to Breiman's "bagging" method when used to aggregate various classifiers.

An Empirical Evaluation of Bagging and Boosting

- Computer ScienceAAAI/IAAI
- 1997

The results clearly show that even though Bagging almost always produces a better classifier than any of its individual component classifiers and is relatively impervious to overfitting, it does not generalize any better than a baseline neural-network ensemble method.