Corpus ID: 5512631

Generalized Boosting Algorithms for Convex Optimization

@article{Grubb2011GeneralizedBA,
  title={Generalized Boosting Algorithms for Convex Optimization},
  author={Alexander Grubb and J. Andrew Bagnell},
  journal={ArXiv},
  year={2011},
  volume={abs/1105.2054}
}
Boosting is a popular way to derive powerful learners from simpler hypothesis classes. Following previous work (Mason et al., 1999; Friedman, 2000) on general boosting frameworks, we analyze gradient-based descent algorithms for boosting with respect to any convex objective and introduce a new measure of weak learner performance into this setting which generalizes existing work. We present the first weak to strong learning guarantees for the existing gradient boosting work for smooth convex… Expand
Gradient Boosting on Stochastic Data Streams
TLDR
This work investigates the problem of adapting batch gradient boosting for minimizing convex loss functions to online setting and presents an algorithm, Streaming Gradient Boosting (SGB), with exponential shrinkage guarantees in the number of weak learners and an adaptation of SGB to optimize non-smooth loss functions. Expand
Rescaled Boosting in Classification
TLDR
This paper aims at developing a new boosting strategy, called rescaled boosting (RBoosting), to accelerate the numerical convergence rate and, consequently, improve learning performances of the original boosting. Expand
Proximal boosting and variants
TLDR
This paper proposes to build upon the proximal point algorithm, when the empirical risk to minimize is not differentiable, in order to introduce a novel boosting approach, called proximal boosting, which exhibits a favorable comparison over gradient boosting regarding convergence rate and prediction accuracy. Expand
Generalized Boosting
TLDR
This work presents a general framework for boosting where it is shown that many popular greedy algorithms for learning deep neural networks (DNNs) can be derived from this framework using function compositions for aggregation, and shows that these algorithms have superior performance over traditional additive boosting algorithms, as well as existing greedy learning techniques for DNNs. Expand
Regularized Gradient Boosting
TLDR
A new algorithm is introduced, called rgb, that directly benefits from these generalization bounds and that, at every boosting round, applies the Structural Risk Minimization principle to search for a base predictor with the best empirical fit versus complexity trade-off. Expand
A new boosting algorithm based on dual averaging scheme
TLDR
This paper examines a particular kind of machine learning algorithm, boosting, whose training process can be viewed as functional coordinate descent on the exponential loss and shows that DABoost, although slower in reducing the training error, in general enjoys a better generalization error than AdaBoost. Expand
A new formulation of gradient boosting
In the setting of regression, the standard formulation of gradient boosting generates a sequence of improvements to a constant model. In this paper, we reformulate gradient boosting such that it isExpand
Soft-max boosting
  • M. Geist
  • Mathematics, Computer Science
  • Machine Learning
  • 2015
TLDR
This paper proposes to replace the usually considered deterministic decision rule by a stochastic one, which allows obtaining a smooth risk (generalizing the expected binary loss, and more generally the cost-sensitive loss) and provides a convergence analysis of the resulting algorithm. Expand
Multi-class ℓ2-Boost with the scoring coding
TLDR
This paper derives a scoring coding scheme using optimal scoring constraints to encode class labels and a simple decoder to recover the true class labels for multi-class boosting algorithms based on coordinate descent and gradient descent to minimize the multi- class l2-loss function. Expand
Greedy Algorithms for Cone Constrained Optimization with Convergence Guarantees
TLDR
This paper considers the intermediate case of optimization over the convex cone, parametrized as the conic hull of a generic atom set, leading to the first principled definitions of non-negative MP algorithms, which give explicit convergence rates and demonstrate excellent empirical performance. Expand
...
1
2
3
4
...

References

SHOWING 1-10 OF 24 REFERENCES
A General Boosting Method and its Application to Learning Ranking Functions for Web Search
We present a general boosting method extending functional gradient boosting to optimize complex loss functions that are encountered in many machine learning problems. Our approach is based onExpand
Greedy function approximation: A gradient boosting machine.
Function estimation/approximation is viewed from the perspective of numerical optimization in function space, rather than parameter space. A connection is made between stagewise additive expansionsExpand
A simpler unified analysis of budget perceptrons
TLDR
A unified analysis of both the Forgetron and the Randomized Budget Perceptron algorithms is proposed by observing that the way in which they remove support vectors can be seen as types of L2-regularization. Expand
On the Convergence of Leveraging
TLDR
An unified convergence analysis of ensemble learning methods including e.g. AdaBoost, Logistic Regression and the Least-Square-Boost algorithm for regression is given, which includes l1-norm regularized cost functions leading to a clean and general way to regularize ensemble learning. Expand
A decision-theoretic generalization of on-line learning and an application to boosting
TLDR
The model studied can be interpreted as a broad, abstract extension of the well-studied on-line prediction model to a general decision-theoretic setting, and the multiplicative weightupdate Littlestone Warmuth rule can be adapted to this model, yielding bounds that are slightly weaker in some cases, but applicable to a considerably more general class of learning problems. Expand
An Efficient Boosting Algorithm for Combining Preferences
TLDR
This work describes and analyze an efficient algorithm called RankBoost for combining preferences based on the boosting approach to machine learning, and gives theoretical results describing the algorithm's behavior both on the training data, and on new test data not seen during training. Expand
Potential Boosters?
TLDR
The question of which potential functions lead to new algorithms that are boosters is examined, and two main results are general sets of conditions on the potential that imply that the resulting algorithm is a booster, while the other implies that the algorithm is not. Expand
The Boosting Approach to Machine Learning An Overview
TLDR
This chapter overviews some of the recent work on boosting including analyses of AdaBoost's training error and generalization error; boosting’s connection to game theory and linear programming; the relationship between boosting and logistic regression; extensions of Ada boost for multiclass classification problems; methods of incorporating human knowledge into boosting; and experimental and applied work using boosting. Expand
A theory of multiclass boosting
TLDR
A broad and general framework is created, within which the optimal requirements on the weak-classifier are identified, as well as design the most effective, in a certain sense, boosting algorithms that assume such requirements. Expand
On the Algorithmic Implementation of Multiclass Kernel-based Vector Machines
TLDR
This paper describes the algorithmic implementation of multiclass kernel-based vector machines using a generalized notion of the margin to multiclass problems, and describes an efficient fixed-point algorithm for solving the reduced optimization problems and proves its convergence. Expand
...
1
2
3
...