On the Algorithmic Implementation of Stochastic Discrimination

@article{Kleinberg2000OnTA,
  title={On the Algorithmic Implementation of Stochastic Discrimination},
  author={Eugene M. Kleinberg},
  journal={IEEE Trans. Pattern Anal. Mach. Intell.},
  year={2000},
  volume={22},
  pages={473-490}
}
  • E. Kleinberg
  • Published 1 May 2000
  • Computer Science
  • IEEE Trans. Pattern Anal. Mach. Intell.
Stochastic discrimination is a general methodology for constructing classifiers appropriate for pattern recognition. It is based on combining arbitrary numbers of very weak components, which are usually generated by some pseudorandom process, and it has the property that the very complex and accurate classifiers produced in this way retain the ability, characteristic of their weak component pieces, to generalize to new data. In fact, it is often observed, in practice, that classifier… 
Building well-performing classifier ensembles : model and decision level combination
TLDR
This thesis reviews the multiple classifier literature and considers the properties an ensemble of classifiers - or collection of subsets - should have in order to be combined successfully, and finds that the framework of Stochastic Discrimination provides a well-defined account of these properties.
A concrete statistical realization of Kleinberg's stochastic discrimination for pattern recognition. Part I. Two-class classification
TLDR
This paper recast SD in a probability-space framework and presents a concrete statistical realization of SD for two-class pattern recognition by introducing near uniformity and weak indiscernibility and analyzes the performance of SD theoretically and explains why SD is overtraining-resistant and why SD has a high convergence rate.
A Mathematically Rigorous Foundation for Supervised Learning
  • E. Kleinberg
  • Computer Science
    Multiple Classifier Systems
  • 2000
TLDR
This paper will provide some algorithmic detail for implementing the general classification method derived from the theory, a method based on classifier combination, and will discuss experimental results comparing its performance to other well-known methods on standard benchmark problems from the U.C. Irvine, and Statlog, collections.
Ensemble feature selection with the simple Bayesian classification
Fields as Limit Functions of Stochastic Discrimination and Their Adaptability
TLDR
For a particular type of elementary function, stochastic discrimination is shown to have an analytic limit function, and two modifications of the limit function are proposed, which lead to classification functions with perfect generalization for high-dimensional parity problems.
On improvement of classification accuracy for stochastic discrimination
TLDR
The proposed improved SD improves standard SD by its capability of achieving higher classification accuracy by showing that the smaller the variance of the discriminant function, the lower the error rate of the classifier.
Random Relational Rules
TLDR
Stochastic discrimination provides a framework for combining arbitrary numbers of weak classifiers in a way where accuracy improves with additional rules, even after maximal accuracy on the training data has been reached.
Learning with Ensembles of Randomized Trees : New Insights
TLDR
A connection with kernel target alignment, a measure of kernel quality, is pointed out, which suggests that randomization is a way to obtain a high alignment, leading to possibly low generalization error.
A complexity framework for combination of classifiers in verification and identification systems
TLDR
This thesis develops a classifier combination framework based on the Bayesian decision theory and introduces the identification model which accounts for the relationships between scores output by one classifier during a single identification trial, which allows the construction of combination methods which consider a whole set of scoresoutput by classifiers in order to derive a combined score for any one class.
Random Forests
TLDR
Internal estimates monitor error, strength, and correlation and these are used to show the response to increasing the number of features used in the forest, and are also applicable to regression.
...
1
2
3
4
5
...

References

SHOWING 1-10 OF 24 REFERENCES
An alternative method of stochastic discrimination with applications to pattern recognition
TLDR
This dissertation introduces an alternative method of performing stochastic discrimination in pattern recognition which differs in several aspects from the original method introduced by Kleinberg, and discusses four variations of the method, each of which uses different variations of Ho's discriminant functions.
Experiments with a New Boosting Algorithm
TLDR
This paper describes experiments carried out to assess how well AdaBoost with and without pseudo-loss, performs on real learning problems and compared boosting to Breiman's "bagging" method when used to aggregate various classifiers.
A Mathematically Rigorous Foundation for Supervised Learning
  • E. Kleinberg
  • Computer Science
    Multiple Classifier Systems
  • 2000
TLDR
This paper will provide some algorithmic detail for implementing the general classification method derived from the theory, a method based on classifier combination, and will discuss experimental results comparing its performance to other well-known methods on standard benchmark problems from the U.C. Irvine, and Statlog, collections.
An overtraining-resistant stochastic modeling method for pattern recognition
We will introduce a generic approach for solving problems in pattern recognition based on the synthesis of accurate multiclass discriminators from large numbers of very inaccurate weak models through
A Decision-Theoretic Generalization of On-Line Learning and an Application to Boosting
TLDR
The model studied can be interpreted as a broad, abstract extension of the well-studied on-line prediction model to a general decision-theoretic setting, and it is shown that the multiplicative weight-update Littlestone?Warmuth rule can be adapted to this model, yielding bounds that are slightly weaker in some cases, but applicable to a considerably more general class of learning problems.
A decision-theoretic generalization of on-line learning and an application to boosting
TLDR
The model studied can be interpreted as a broad, abstract extension of the well-studied on-line prediction model to a general decision-theoretic setting, and the multiplicative weightupdate Littlestone Warmuth rule can be adapted to this model, yielding bounds that are slightly weaker in some cases, but applicable to a considerably more general class of learning problems.
Machine learning
TLDR
Machine learning addresses many of the same research questions as the fields of statistics, data mining, and psychology, but with differences of emphasis.
The Random Subspace Method for Constructing Decision Forests
  • T. Ho
  • Computer Science
    IEEE Trans. Pattern Anal. Mach. Intell.
  • 1998
TLDR
A method to construct a decision tree based classifier is proposed that maintains highest accuracy on training data and improves on generalization accuracy as it grows in complexity.
Programs for Machine Learning
TLDR
In his new book, C4.5: Programs for Machine Learning, Quinlan has put together a definitive, much needed description of his complete system, including the latest developments, which will be a welcome addition to the library of many researchers and students.
Stochastic discrimination
  • E. Kleinberg
  • Mathematics
    Annals of Mathematics and Artificial Intelligence
  • 2005
TLDR
A general method is introduced for separating points in multidimensional spaces through the use of stochastic processes, called Stochastic discrimination.
...
1
2
3
...