• Corpus ID: 74721

Robust Boosting via Convex Optimization: Theory and Applications

@inproceedings{Rtsch2007RobustBV,
  title={Robust Boosting via Convex Optimization: Theory and Applications},
  author={Gunnar R{\"a}tsch},
  year={2007}
}
In this work we consider statistical learning problems. A learning machine aims to extract information from a set of training examples such that it is able to predict the associated label on unseen examples. We consider the case where the resulting classification or regression rule is a combination of simple rules – also called base hypotheses. The so-called boosting algorithms iteratively find a weighted linear combination of base hypotheses that predict well on unseen data. We address the… 
Boosting Algorithms for Maximizing the Soft Margin
TLDR
This work presents a novel boosting algorithm, called SoftBoost, designed for sets of binary labeled examples that are not necessarily separable by convex combinations of base hypotheses, that produces a convex combination of hypotheses whose soft margin is within δ of its maximum.
Ensembles of Partially Trained SVMs with Multiplicative Updates
TLDR
This paper shows that the multiplicative update of SVM can be formulated as a Bregman projection problem, and the learning rate can then be adapted automatically, and it is shown that the proposed ensemble has efficient training and comparable or even better accuracy than the best-tuned soft-margin SVM.
On the Current State of Research in Explaining Ensemble Performance Using Margins
TLDR
Several techniques are proposed and evidence suggesting that the generalization error of a voting classifier might be reduced by increasing the mean and decreasing the variance of the margins is provided, suggesting the current state of research in explaining ensemble performance holds.
Reducing the Overfitting of Adaboost by Controlling its Data Distribution Skewness
TLDR
This paper derives two new regularized AdaBoost algorithms, referred to as AdaBoostKL and AdaBoostNorm2, respectively, from two smooth convex penalty functions, based on Kullback–Leibler divergence and l2 norm, and proves that their algorithms perform stage-wise gradient descent on a cost function, defined in the domain of their associated soft margins.
Entropy Regularized LPBoost
TLDR
By simply adding a relative entropy regularization to the linear objective of LPBoost, this paper arrives at the Entropy Regularized LPBoost algorithm for which it is shown that this algorithm can be proved to have a logarithmic iteration bound.
A Tutorial Review of RKHS Methods in Machine Learning
Over the last ten years, estimation and learning methods utilizing positive definite kernels have become rather popular, particularly in machine learning. Since these methods have a stronger
Analysis of boosting algorithms using the smooth margin function
We introduce a useful tool for analyzing boosting algorithms called the "smooth margin function," a differentiable approximation of the usual margin for boosting algorithms. We present two boosting
A Dynamic AdaBoost Algorithm With Adaptive Changes of Loss Function
TLDR
DAdaBoost can effectively prevent AdaBoost from overfitting, and Hoeffding inequality is adopted as a statistical tool to divide training samples into reliable samples and temporary unreliable samples.
Submitted to the Annals of Statistics ANALYSIS OF BOOSTING ALGORITHMS USING THE SMOOTH MARGIN FUNCTION ∗ By
Columbia University and Princeton University We introduce a useful tool for analyzing boosting algorithms called the “smooth margin function”, a differentiable approximation of the usual margin for
Approximate Reduction from AUC Maximization to 1-Norm Soft Margin Optimization
TLDR
This paper shows that the hard margin case is approximately reduced to a soft margin optimization problem over p + n instances for which the resulting linear classifier is guaranteed to have a certain margin over pairs.
...
1
2
3
4
5
...

References

SHOWING 1-10 OF 255 REFERENCES
Boosting in the Limit: Maximizing the Margin of Learned Ensembles
TLDR
The crucial question as to why boosting works so well in practice, and how to further improve upon it, remains mostly open, and it is concluded that no simple version of the minimum-margin story can be complete.
Improved Generalization Through Explicit Optimization of Margins
TLDR
A theorem bounding the generalization performance of convex combinations in terms of general cost functions of the margin is proved, in contrast to previous results, which were stated in Terms of the particular cost function sgn(θ − margin).
Special Invited Paper-Additive logistic regression: A statistical view of boosting
Boosting is one of the most important recent developments in classification methodology. Boosting works by sequentially applying a classification algorithm to reweighted versions of the training data
On weak base hypotheses and their implications for boosting regression and classification
When studying the training error and the prediction error for boosting, it is often assumed that the hypotheses returned by the base learner are weakly accurate, or are able to beat a random guesser
Linear Programming Boosting via Column Generation
TLDR
It is proved that for classification, minimizing the 1-norm soft margin error function directly optimizes a generalization error bound and is competitive in quality and computational cost to AdaBoost.
Constructing Boosting Algorithms from SVMs: An Application to One-Class Classification
We show via an equivalence of mathematical programs that a support vector (SV) algorithm can be translated into an equivalent boosting-like algorithm and vice versa. We exemplify this translation
Robust Bounds on Generalization from the Margin Distribution
A number of results have bounded generalization of a classifier in terms of its margin on the training points. There has been some debate about whether the minimum margin is the best measure of the
A decision-theoretic generalization of on-line learning and an application to boosting
TLDR
The model studied can be interpreted as a broad, abstract extension of the well-studied on-line prediction model to a general decision-theoretic setting, and the multiplicative weightupdate Littlestone Warmuth rule can be adapted to this model, yielding bounds that are slightly weaker in some cases, but applicable to a considerably more general class of learning problems.
Experiments with a New Boosting Algorithm
TLDR
This paper describes experiments carried out to assess how well AdaBoost with and without pseudo-loss, performs on real learning problems and compared boosting to Breiman's "bagging" method when used to aggregate various classifiers.
An Empirical Evaluation of Bagging and Boosting
TLDR
The results clearly show that even though Bagging almost always produces a better classifier than any of its individual component classifiers and is relatively impervious to overfitting, it does not generalize any better than a baseline neural-network ensemble method.
...
1
2
3
4
5
...