# Random Forests and Adaptive Nearest Neighbors

@article{Lin2006RandomFA, title={Random Forests and Adaptive Nearest Neighbors}, author={Yi Lin and Yongho Jeon}, journal={Journal of the American Statistical Association}, year={2006}, volume={101}, pages={578 - 590} }

In this article we study random forests through their connection with a new framework of adaptive nearest-neighbor methods. We introduce a concept of potential nearest neighbors (k-PNNs) and show that random forests can be viewed as adaptively weighted k-PNN methods. Various aspects of random forests can be studied from this perspective. We study the effect of terminal node sizes on the prediction accuracy of random forests. We further show that random forests with adaptive splitting schemes…

## 422 Citations

Banzhaf Random Forests

- Computer ScienceArXiv
- 2015

Experiments show that the proposed novel random forests algorithm, named Banzhaf random forests (BRF), is competitive with state-of-the-art classifiers and dramatically outperforms previous consistent random forests.

Towards Convergence Rate Analysis of Random Forests for Classification

- Computer Science, MathematicsNeurIPS
- 2020

This work presents the first finite-sample rate O(n−1/(8d+2)) on the convergence of pure random forests for classification, which can be improved to be of O( n−1/3.87d-2) by considering the midpoint splitting mechanism.

Learning with random forests

- Computer Science
- 2015

The link between infinite forests and finite forests is focused on, aiming at narrowing the gap between theory and practice, and the importance of subsampling is stressed to demonstrate the consistency of the unpruned Breiman's forests.

Asymptotic Theory for Random Forests

- Computer Science, Mathematics
- 2014

A random forest model based on subsampling is analyzed, and it is shown that random forest predictions are asymptotically normal provided that the subsample size s scales as s(n)/n = o(log(n)^{-d}), where n is the number of training examples and d is theNumber of features.

When do random forests fail?

- Computer ScienceNeurIPS
- 2018

This paper considers various tree constructions and examines how the choice of parameters affects the generalization error of the resulting random forests as the sample size goes to infinity, and shows that trees that have good performance in nearest-neighbor search can be a poor choice for random forests.

Random Forests and Kernel Methods

- Computer ScienceIEEE Transactions on Information Theory
- 2016

It is shown that by slightly modifying their definition, random forests can be rewritten as kernel methods (called KeRF for kernel based on random forests) which are more interpretable and easier to analyze.

Analysis of purely random forests bias

- Computer Science, MathematicsArXiv
- 2014

Under some regularity assumptions on the regression function, it is shown that the bias of an infinite forest decreases at a faster rate (with respect to the size of each tree) than a single tree, and infinite forests attain a strictly better risk rate than single trees.

Random Uniform Forests

- Computer Science
- 2015

The main motivation of Random Uniform Forests is to be more weakly dependent to the data than Random Forests while giving similar performance and inheriting of all their theoretical properties.

Generalized random forests

- Computer Science, MathematicsThe Annals of Statistics
- 2019

A flexible, computationally efficient algorithm for growing generalized random forests, an adaptive weighting function derived from a forest designed to express heterogeneity in the specified quantity of interest, and an estimator for their asymptotic variance that enables valid confidence intervals are proposed.

Random KNN feature selection - a fast and stable alternative to Random Forests

- Computer ScienceBMC Bioinformatics
- 2011

This work proposes RKNN-FS, an innovative feature selection procedure based on Random KNN (RKNN), a novel generalization of traditional nearest-neighbor modeling that is significantly more stable and more robust than Random Forests for feature selection when the input data are noisy and/or unbalanced.

## References

SHOWING 1-10 OF 43 REFERENCES

Random Forests

- Computer ScienceMachine Learning
- 2004

Internal estimates monitor error, strength, and correlation and these are used to show the response to increasing the number of features used in the forest, and are also applicable to regression.

Locally Adaptive Metric Nearest-Neighbor Classification

- Computer ScienceIEEE Trans. Pattern Anal. Mach. Intell.
- 2002

A chi-squared distance analysis is used to compute a flexible metric for producing neighborhoods that are highly adaptive to query locations and the class conditional probabilities are smoother in the modified neighborhoods, whereby better classification performance can be achieved.

Discriminant Adaptive Nearest Neighbor Classification

- Computer ScienceIEEE Trans. Pattern Anal. Mach. Intell.
- 1996

A locally adaptive form of nearest neighbor classification is proposed to try to finesse this curse of dimensionality, and a method for global dimension reduction is proposed, that combines local dimension information.

Special Invited Paper-Additive logistic regression: A statistical view of boosting

- Computer Science
- 2000

This work shows that this seemingly mysterious phenomenon of boosting can be understood in terms of well-known statistical principles, namely additive modeling and maximum likelihood, and develops more direct approximations and shows that they exhibit nearly identical results to boosting.

The Random Subspace Method for Constructing Decision Forests

- Computer ScienceIEEE Trans. Pattern Anal. Mach. Intell.
- 1998

A method to construct a decision tree based classifier is proposed that maintains highest accuracy on training data and improves on generalization accuracy as it grows in complexity.

Shape Quantization and Recognition with Randomized Trees

- Computer ScienceNeural Computation
- 1997

A new approach to shape recognition based on a virtually infinite family of binary features (queries) of the image data, designed to accommodate prior information about shape invariance and regularity, and a comparison with artificial neural networks methods is presented.

SOME INFINITY THEORY FOR PREDICTOR ENSEMBLES

- Computer Science
- 2000

It is shown that the simplest kind of trees are complete in D-dimensional space if the number of terminal nodes T is greater than D and that the Adaboost minimization algorithm gives an ensemble converging to the Bayes risk.

Experiments with a New Boosting Algorithm

- Computer ScienceICML
- 1996

This paper describes experiments carried out to assess how well AdaBoost with and without pseudo-loss, performs on real learning problems and compared boosting to Breiman's "bagging" method when used to aggregate various classifiers.

Boosting a weak learning algorithm by majority

- Computer ScienceCOLT '90
- 1990

An algorithm for improving the accuracy of algorithms for learning binary concepts by combining a large number of hypotheses, each of which is generated by training the given learning algorithm on a different set of examples, is presented.

Greedy function approximation: A gradient boosting machine.

- Computer Science
- 2001

A general gradient descent boosting paradigm is developed for additive expansions based on any fitting criterion, and specific algorithms are presented for least-squares, least absolute deviation, and Huber-M loss functions for regression, and multiclass logistic likelihood for classification.