Random Forests and Adaptive Nearest Neighbors

@article{Lin2006RandomFA,
  title={Random Forests and Adaptive Nearest Neighbors},
  author={Yi Lin and Yongho Jeon},
  journal={Journal of the American Statistical Association},
  year={2006},
  volume={101},
  pages={578 - 590}
}
  • Yi Lin, Yongho Jeon
  • Published 1 June 2006
  • Computer Science
  • Journal of the American Statistical Association
In this article we study random forests through their connection with a new framework of adaptive nearest-neighbor methods. We introduce a concept of potential nearest neighbors (k-PNNs) and show that random forests can be viewed as adaptively weighted k-PNN methods. Various aspects of random forests can be studied from this perspective. We study the effect of terminal node sizes on the prediction accuracy of random forests. We further show that random forests with adaptive splitting schemes… 
Banzhaf Random Forests
TLDR
Experiments show that the proposed novel random forests algorithm, named Banzhaf random forests (BRF), is competitive with state-of-the-art classifiers and dramatically outperforms previous consistent random forests.
Towards Convergence Rate Analysis of Random Forests for Classification
TLDR
This work presents the first finite-sample rate O(n−1/(8d+2)) on the convergence of pure random forests for classification, which can be improved to be of O( n−1/3.87d-2) by considering the midpoint splitting mechanism.
Learning with random forests
TLDR
The link between infinite forests and finite forests is focused on, aiming at narrowing the gap between theory and practice, and the importance of subsampling is stressed to demonstrate the consistency of the unpruned Breiman's forests.
Asymptotic Theory for Random Forests
TLDR
A random forest model based on subsampling is analyzed, and it is shown that random forest predictions are asymptotically normal provided that the subsample size s scales as s(n)/n = o(log(n)^{-d}), where n is the number of training examples and d is theNumber of features.
When do random forests fail?
TLDR
This paper considers various tree constructions and examines how the choice of parameters affects the generalization error of the resulting random forests as the sample size goes to infinity, and shows that trees that have good performance in nearest-neighbor search can be a poor choice for random forests.
Random Forests and Kernel Methods
  • Erwan Scornet
  • Computer Science
    IEEE Transactions on Information Theory
  • 2016
TLDR
It is shown that by slightly modifying their definition, random forests can be rewritten as kernel methods (called KeRF for kernel based on random forests) which are more interpretable and easier to analyze.
Analysis of purely random forests bias
TLDR
Under some regularity assumptions on the regression function, it is shown that the bias of an infinite forest decreases at a faster rate (with respect to the size of each tree) than a single tree, and infinite forests attain a strictly better risk rate than single trees.
Random Uniform Forests
TLDR
The main motivation of Random Uniform Forests is to be more weakly dependent to the data than Random Forests while giving similar performance and inheriting of all their theoretical properties.
Generalized random forests
TLDR
A flexible, computationally efficient algorithm for growing generalized random forests, an adaptive weighting function derived from a forest designed to express heterogeneity in the specified quantity of interest, and an estimator for their asymptotic variance that enables valid confidence intervals are proposed.
Random KNN feature selection - a fast and stable alternative to Random Forests
TLDR
This work proposes RKNN-FS, an innovative feature selection procedure based on Random KNN (RKNN), a novel generalization of traditional nearest-neighbor modeling that is significantly more stable and more robust than Random Forests for feature selection when the input data are noisy and/or unbalanced.
...
1
2
3
4
5
...

References

SHOWING 1-10 OF 43 REFERENCES
Random Forests
TLDR
Internal estimates monitor error, strength, and correlation and these are used to show the response to increasing the number of features used in the forest, and are also applicable to regression.
Locally Adaptive Metric Nearest-Neighbor Classification
TLDR
A chi-squared distance analysis is used to compute a flexible metric for producing neighborhoods that are highly adaptive to query locations and the class conditional probabilities are smoother in the modified neighborhoods, whereby better classification performance can be achieved.
Discriminant Adaptive Nearest Neighbor Classification
TLDR
A locally adaptive form of nearest neighbor classification is proposed to try to finesse this curse of dimensionality, and a method for global dimension reduction is proposed, that combines local dimension information.
Special Invited Paper-Additive logistic regression: A statistical view of boosting
TLDR
This work shows that this seemingly mysterious phenomenon of boosting can be understood in terms of well-known statistical principles, namely additive modeling and maximum likelihood, and develops more direct approximations and shows that they exhibit nearly identical results to boosting.
The Random Subspace Method for Constructing Decision Forests
  • T. Ho
  • Computer Science
    IEEE Trans. Pattern Anal. Mach. Intell.
  • 1998
TLDR
A method to construct a decision tree based classifier is proposed that maintains highest accuracy on training data and improves on generalization accuracy as it grows in complexity.
Shape Quantization and Recognition with Randomized Trees
TLDR
A new approach to shape recognition based on a virtually infinite family of binary features (queries) of the image data, designed to accommodate prior information about shape invariance and regularity, and a comparison with artificial neural networks methods is presented.
SOME INFINITY THEORY FOR PREDICTOR ENSEMBLES
TLDR
It is shown that the simplest kind of trees are complete in D-dimensional space if the number of terminal nodes T is greater than D and that the Adaboost minimization algorithm gives an ensemble converging to the Bayes risk.
Experiments with a New Boosting Algorithm
TLDR
This paper describes experiments carried out to assess how well AdaBoost with and without pseudo-loss, performs on real learning problems and compared boosting to Breiman's "bagging" method when used to aggregate various classifiers.
Boosting a weak learning algorithm by majority
TLDR
An algorithm for improving the accuracy of algorithms for learning binary concepts by combining a large number of hypotheses, each of which is generated by training the given learning algorithm on a different set of examples, is presented.
Greedy function approximation: A gradient boosting machine.
TLDR
A general gradient descent boosting paradigm is developed for additive expansions based on any fitting criterion, and specific algorithms are presented for least-squares, least absolute deviation, and Huber-M loss functions for regression, and multiclass logistic likelihood for classification.
...
1
2
3
4
5
...