Support Vector Machines and the Bayes Rule in Classification

@article{Lin2004SupportVM,
  title={Support Vector Machines and the Bayes Rule in Classification},
  author={Yi Lin},
  journal={Data Mining and Knowledge Discovery},
  year={2004},
  volume={6},
  pages={259-275}
}
  • Yi Lin
  • Published 1 July 2002
  • Computer Science
  • Data Mining and Knowledge Discovery
The Bayes rule is the optimal classification rule if the underlying distribution of the data is known. In practice we do not know the underlying distribution, and need to “learn” classification rules from the data. One way to derive classification rules in practice is to implement the Bayes rule approximately by estimating an appropriate classification function. Traditional statistical methods use estimated log odds ratio as the classification function. Support vector machines (SVMs) are one… 

Statistical Properties and Adaptive Tuning of Support Vector Machines

TLDR
An approach to adaptively tuning the smoothing parameter(s) in the SVMs is described, based on the generalized approximate cross validation (GACV), which is an easily computable proxy of the GCKL.

A Note on Margin-based Loss Functions in Classification by

TLDR
It is shown that the hinge loss is the tightest convex upper bound of the misclassification loss, and the Fisher consistency of margin-based loss functions often leads to consistency and rate of convergence (to the Bayes optimal risk) results under general conditions.

A Note on Margin-based Loss Functions in Classification by Yi Lin A Note on Margin-based Loss Functions in Classification

TLDR
It is shown that the hinge loss is the tightest convex upper bound of the misclassification loss, and the Fisher consistency of margin-based loss functions often leads to consistency and rate of convergence (to the Bayes optimal risk) results under general conditions.

Statistical performance of support vector machines

TLDR
The main result shows that it is possible to obtain fast rates of convergence for SVMs and builds on the observation made by other authors that the SVM can be viewed as a statistical regularization procedure.

Comparing naive Bayes, decision trees, and SVM with AUC and accuracy

TLDR
It is proved that AUC is, in general, a better measure (defined precisely) than accuracy for evaluating performance of learning algorithms.

NORM SUPPORT VECTOR MACHINE

TLDR
It is shown that the F∞-norm SVM is equivalent to a linear programming problem and can be efficiently solved using standard techniques and enjoys competitive performance when compared with the 1-norm and 2-norm SVMs.

Large‐margin classification with multiple decision rules

TLDR
This paper proposes a spectrum of statistical learning problems that span the hard and soft classification tasks based on fitting multiple decision rules to the data, and reveals a novel collection of learning tasks of increasing complexity.

Support Vector Machines with Applications

TLDR
This paper is intended as an introduction to SVMs and their applications, emphasizing their key features, and some algorithmic extensions and illustrative real-world applications of SVMs are shown.

Multiclass Probability Estimation With Support Vector Machines

TLDR
This article proposes a simple yet effective framework to endow kernel SVMs with the feature of multiclass probability estimation and demonstrates competitive performance of the new estimator when compared with existing methods such as multiple logistic regression, linear discrimination analysis, tree-based methods, and random forest, under various classification settings.
...

References

SHOWING 1-10 OF 19 REFERENCES

Support Vector Machines for Classification in Nonstandard Situations

TLDR
This paper explains why the standard support vectors machine is not suitable for the nonstandard situation, and introduces a simple procedure for adapting the support vector machine methodology to the non standard situation.

On Bias, Variance, 0/1—Loss, and the Curse-of-Dimensionality

  • J. Friedman
  • Mathematics
    Data Mining and Knowledge Discovery
  • 2004
TLDR
This work candramatically mitigate the effect of the bias associated with some simpleestimators like “naive” Bayes, and the bias induced by the curse-of-dimensionality on nearest-neighbor procedures.

Robust Bounds on Generalization from the Margin Distribution

TLDR
It is shown that a slight generalization of their construction can be used to give a pac style bound on the tail of the distribution of the generalization errors that arise from a given sample size.

Support-Vector Networks

TLDR
High generalization ability of support-vector networks utilizing polynomial input transformations is demonstrated and the performance of the support- vector network is compared to various classical learning algorithms that all took part in a benchmark study of Optical Character Recognition.

Tensor product space ANOVA models in multivariate function estimation

  • Yi Lin
  • Mathematics, Computer Science
  • 1998
TLDR
The main result is that, in a variety of general nonparametric problems, under general conditions, the rate of convergence for the penalized likelihood estimator in the TPS-ANOVA model is O([n(logn)1 r] 2m 2m+1 ) when the smoothing parameter is appropriately chosen, which means that curse of dimensionality is overcome by the tensor product space ANOVA models.

Tensor product space ANOVA models

  • Yi Lin
  • Mathematics, Computer Science
  • 2000
TLDR
The quick optimal rate of the TPS-ANOVA model makes it very preferable in high-dimensional function estimation, and many properties of the tensor product space of Sobolev-Hilbert spaces are given.

Asymptotic Analysis of Penalized Likelihood and Related Estimators

A general approach to the first order asymptotic analysis ofpenalized likelihood and related estimators is described. The method gives expansions for the systematic and random error. Asymptotic

A Unified Framework for Regularization Networks and Support Vector Machines

TLDR
This work presents regularization Networks and Support Vector Machines in a unified framework in the context of Vapnik''s theory of statistical learning which provides a general foundation for the learning problem, combining functional analysis and statistics.

What is a support vector machine?

Support vector machines (SVMs) are becoming popular in a wide variety of biological applications. But, what exactly are SVMs and how do they work? And what are their most promising applications in

The Nature of Statistical Learning Theory

  • V. Vapnik
  • Computer Science
    Statistics for Engineering and Information Science
  • 2000
Setting of the learning problem consistency of learning processes bounds on the rate of convergence of learning processes controlling the generalization ability of learning processes constructing