# On the Algorithmic Implementation of Stochastic Discrimination

@article{Kleinberg2000OnTA, title={On the Algorithmic Implementation of Stochastic Discrimination}, author={Eugene M. Kleinberg}, journal={IEEE Trans. Pattern Anal. Mach. Intell.}, year={2000}, volume={22}, pages={473-490} }

Stochastic discrimination is a general methodology for constructing classifiers appropriate for pattern recognition. It is based on combining arbitrary numbers of very weak components, which are usually generated by some pseudorandom process, and it has the property that the very complex and accurate classifiers produced in this way retain the ability, characteristic of their weak component pieces, to generalize to new data. In fact, it is often observed, in practice, that classifier…

## Figures from this paper

## 119 Citations

Building well-performing classifier ensembles : model and decision level combination

- Computer Science
- 2010

This thesis reviews the multiple classifier literature and considers the properties an ensemble of classifiers - or collection of subsets - should have in order to be combined successfully, and finds that the framework of Stochastic Discrimination provides a well-defined account of these properties.

A concrete statistical realization of Kleinberg's stochastic discrimination for pattern recognition. Part I. Two-class classification

- Computer Science
- 2002

This paper recast SD in a probability-space framework and presents a concrete statistical realization of SD for two-class pattern recognition by introducing near uniformity and weak indiscernibility and analyzes the performance of SD theoretically and explains why SD is overtraining-resistant and why SD has a high convergence rate.

A Mathematically Rigorous Foundation for Supervised Learning

- Computer ScienceMultiple Classifier Systems
- 2000

This paper will provide some algorithmic detail for implementing the general classification method derived from the theory, a method based on classifier combination, and will discuss experimental results comparing its performance to other well-known methods on standard benchmark problems from the U.C. Irvine, and Statlog, collections.

Ensemble feature selection with the simple Bayesian classification

- Computer ScienceInf. Fusion
- 2003

Fields as Limit Functions of Stochastic Discrimination and Their Adaptability

- MathematicsNeural Computation
- 2002

For a particular type of elementary function, stochastic discrimination is shown to have an analytic limit function, and two modifications of the limit function are proposed, which lead to classification functions with perfect generalization for high-dimensional parity problems.

On improvement of classification accuracy for stochastic discrimination

- MathematicsIEEE Trans. Syst. Man Cybern. Part B
- 2005

The proposed improved SD improves standard SD by its capability of achieving higher classification accuracy by showing that the smaller the variance of the discriminant function, the lower the error rate of the classifier.

Random Relational Rules

- Computer Science
- 2006

Stochastic discrimination provides a framework for combining arbitrary numbers of weak classifiers in a way where accuracy improves with additional rules, even after maximal accuracy on the training data has been reached.

Learning with Ensembles of Randomized Trees : New Insights

- Computer ScienceECML/PKDD
- 2010

A connection with kernel target alignment, a measure of kernel quality, is pointed out, which suggests that randomization is a way to obtain a high alignment, leading to possibly low generalization error.

A complexity framework for combination of classifiers in verification and identification systems

- Computer Science
- 2006

This thesis develops a classifier combination framework based on the Bayesian decision theory and introduces the identification model which accounts for the relationships between scores output by one classifier during a single identification trial, which allows the construction of combination methods which consider a whole set of scoresoutput by classifiers in order to derive a combined score for any one class.

Random Forests

- Computer ScienceMachine Learning
- 2004

Internal estimates monitor error, strength, and correlation and these are used to show the response to increasing the number of features used in the forest, and are also applicable to regression.

## References

SHOWING 1-10 OF 24 REFERENCES

An alternative method of stochastic discrimination with applications to pattern recognition

- Computer Science
- 1995

This dissertation introduces an alternative method of performing stochastic discrimination in pattern recognition which differs in several aspects from the original method introduced by Kleinberg, and discusses four variations of the method, each of which uses different variations of Ho's discriminant functions.

Experiments with a New Boosting Algorithm

- Computer ScienceICML
- 1996

This paper describes experiments carried out to assess how well AdaBoost with and without pseudo-loss, performs on real learning problems and compared boosting to Breiman's "bagging" method when used to aggregate various classifiers.

A Mathematically Rigorous Foundation for Supervised Learning

- Computer ScienceMultiple Classifier Systems
- 2000

This paper will provide some algorithmic detail for implementing the general classification method derived from the theory, a method based on classifier combination, and will discuss experimental results comparing its performance to other well-known methods on standard benchmark problems from the U.C. Irvine, and Statlog, collections.

An overtraining-resistant stochastic modeling method for pattern recognition

- Computer Science
- 1996

We will introduce a generic approach for solving problems in pattern recognition based on the synthesis of accurate multiclass discriminators from large numbers of very inaccurate weak models through…

A Decision-Theoretic Generalization of On-Line Learning and an Application to Boosting

- Computer ScienceCOLT 1997
- 1997

The model studied can be interpreted as a broad, abstract extension of the well-studied on-line prediction model to a general decision-theoretic setting, and it is shown that the multiplicative weight-update Littlestone?Warmuth rule can be adapted to this model, yielding bounds that are slightly weaker in some cases, but applicable to a considerably more general class of learning problems.

A decision-theoretic generalization of on-line learning and an application to boosting

- Computer ScienceEuroCOLT
- 1995

The model studied can be interpreted as a broad, abstract extension of the well-studied on-line prediction model to a general decision-theoretic setting, and the multiplicative weightupdate Littlestone Warmuth rule can be adapted to this model, yielding bounds that are slightly weaker in some cases, but applicable to a considerably more general class of learning problems.

Machine learning

- Computer ScienceCSUR
- 1996

Machine learning addresses many of the same research questions as the fields of statistics, data mining, and psychology, but with differences of emphasis.

The Random Subspace Method for Constructing Decision Forests

- Computer ScienceIEEE Trans. Pattern Anal. Mach. Intell.
- 1998

A method to construct a decision tree based classifier is proposed that maintains highest accuracy on training data and improves on generalization accuracy as it grows in complexity.

Programs for Machine Learning

- Computer Science
- 1994

In his new book, C4.5: Programs for Machine Learning, Quinlan has put together a definitive, much needed description of his complete system, including the latest developments, which will be a welcome addition to the library of many researchers and students.

Stochastic discrimination

- MathematicsAnnals of Mathematics and Artificial Intelligence
- 2005

A general method is introduced for separating points in multidimensional spaces through the use of stochastic processes, called Stochastic discrimination.