A tutorial on support vector regression

@article{Smola2004ATO,
  title={A tutorial on support vector regression},
  author={Alex Smola and Bernhard Sch{\"o}lkopf},
  journal={Statistics and Computing},
  year={2004},
  volume={14},
  pages={199-222}
}
In this tutorial we give an overview of the basic ideas underlying Support Vector (SV) machines for function estimation. Furthermore, we include a summary of currently used algorithms for training SV machines, covering both the quadratic (or convex) programming part and advanced methods for dealing with large datasets. Finally, we mention some modifications and extensions that have been applied to the standard SV algorithm, and discuss the aspect of regularization from a SV perspective. 

Support Vector Selection for Regression Machines

The orthogonal least-squares method is adopted to evaluate the support vectors based on their error reduction ratios and a simpler model is obtained which helps avoid the over-fitting problem.

Kernel methods: a survey of current techniques

A general formulation for support vector machines

  • Wei ChuS. KeerthiC. Ong
  • Computer Science
    Proceedings of the 9th International Conference on Neural Information Processing, 2002. ICONIP '02.
  • 2002
This paper derives a general formulation of support vector machines for classification and regression respectively as a patch of L/sub 1/ and L/ sub 2/ soft margin loss functions for classifier and soft insensitive loss function is introduced as the generalization of popular loss function for regression.

Complex support vector regression

This work employs the recently presented Wirtinger's calculus on complex RKHS to compute the Lagrangian and derive the dual problem, and proves that this approach is equivalent with solving two real SVR problems exploiting a specific real kernel.

Computing the Solution Path for the Regularized Support Vector Regression

An algorithm is derived that computes the entire solution path of the support vector regression, with the same computational cost as fitting one SVR model, which allows convenient selection of the regularization parameter.

Efficient Computation and Model Selection for the Support Vector Regression

An algorithm is derived that computes the entire solution path of the support vector regression (SVR) and an unbiased estimate for the degrees of freedom of the SVR model is proposed, which allows convenient selection of the regularization parameter.

Inference for Support Vector Regression under ℓ1 Regularization

This work provides large-sample distribution theory for support vector regression with l1-norm along with error bars for the SVR regression coefficients and proposes an alternative large- sample inference method based on the inversion of a novel test statistic that displays competitive power properties and does not depend on the choice of a tuning parameter.

A Note on Least Squares Support Vector Machines

An improved conjugate gradient scheme is proposed for solving the optimization problems in LS-SVM, and an improved SMO algorithm is put forward for the general unconstrained quadratic programming problems which is the case of LS- SVM without the bias term.

Optimal Selection of the Regression Kernel Matrix with Semidefinite Programming

Preliminary experimental results are presented for which the optimal kernel matrix for support vector machine regression is retrieved.
...

References

SHOWING 1-10 OF 223 REFERENCES

A Tutorial on Support Vector Regression Produced as Part of the Esprit Working Group in Neural and Computational Learning Ii, Neurocolt2 27150

This tutorial gives an overview of the basic ideas underlying Support Vector machines for regression and function estimation, and includes a summary of currently used algorithms for training SV machines, covering both the quadratic programming part and advanced methods for dealing with large datasets.

The Relevance Vector Machine

The Relevance Vector Machine is introduced, a Bayesian treatment of a generalised linear model of identical functional form to the SVM, and examples demonstrate that for comparable generalisation performance, the RVM requires dramatically fewer kernel functions.

Regression estimation with support vector learning machines

Support Vector Learning Machines are able to construct spline approximations of given data independently from the number of input-dimensions regarding complexity during training and with only linear complexity for reconstruction-compared to exponential complexity in conventional methods.

New Support Vector Algorithms

A new class of support vector algorithms for regression and classification that eliminates one of the other free parameters of the algorithm: the accuracy parameter in the regression case, and the regularization constant C in the classification case.

Support Vector Regression Machines

This work compares support vector regression (SVR) with a committee regression technique (bagging) based on regression trees and ridge regression done in feature space and expects that SVR will have advantages in high dimensionality space because SVR optimization does not depend on the dimensionality of the input space.

The Support Vector Method

  • V. Vapnik
  • Computer Science, Mathematics
    ICANN
  • 1997
The general idea of the Support Vector method is described and theorems demonstrating that the generalization ability of the SV method is based on factors which classical statistics do not take into account are presented.

Bounds on Error Expectation for Support Vector Machines

It is proved that the value of the span is always smaller (and can be much smaller) than the diameter of the smallest sphere containing the support vectors, used in previous bounds.

Probabilistic kernel regression models

A class of exible conditional probability models and techniques for classi cation regression problems that comes from the use of kernel functions as in support vector machines and the generality from dual formulations of stan dard regression models is introduced.

On the Optimal Parameter Choice for v-Support Vector Machines

  • Ingo Steinwart
  • Computer Science
    IEEE Trans. Pattern Anal. Mach. Intell.
  • 2003
It turns out that /spl nu/ should be a close upper estimate of twice the optimal Bayes risk provided that the classifier uses a so-called universal kernel such as the Gaussian RBF kernel.

The analysis of decomposition methods for support vector machines

This paper connects this method to projected gradient methods and provides theoretical proofs for a version of decomposition methods and shows that this convergence proof is valid for general decomposition Methods if their working set selection meets a simple requirement.
...