A comparative study of ordinary cross-validation, v-fold cross-validation and the repeated learning-testing methods

@article{Burman1989ACS,
  title={A comparative study of ordinary cross-validation, v-fold cross-validation and the repeated learning-testing methods},
  author={Prabir Burman},
  journal={Biometrika},
  year={1989},
  volume={76},
  pages={503-514}
}
  • P. Burman
  • Published 1 September 1989
  • Biology, Business
  • Biometrika
SUMMARY Concepts of v-fold cross-validation and repeated learning-testing methods have been introduced here. In many problems, these methods are computationally much less expensive than ordinary cross-validation and can be used in its place. A comparative study of these three methods has been carried out in detail. 

Tables from this paper

Some Issues in Cross-Validation
TLDR
A new type of cross- validation is proposed here for model selection problems when the data is generated by a stationary process, which is an off-shoot of both ordinary cross-validation and v-fold cross- validation.
An assessment of ten-fold and Monte Carlo cross validations for time series forecasting
  • Rigoberto Fonseca, P. Gómez-Gil
  • Computer Science
    2013 10th International Conference on Electrical Engineering, Computing Science and Automatic Control (CCE)
  • 2013
TLDR
Experimental results, using time series of the NN3 tournament, found that Monte Carlo cross validation is more stable than ten-fold cross validation for selecting the best forecasting model.
Multiple predicting K-fold cross-validation for model selection
TLDR
This study proposes a new CV method that uses folds of the data for model validation, while the other fold is for model construction, and provides predicted values for each observation to reduce variation in the assessment due to the averaging.
On the Use of K-Fold Cross-Validation to Choose Cutoff Values and Assess the Performance of Predictive Models in Stepwise Regression
TLDR
This paper addresses a methodological technique of leave-many-out cross-validation for choosing cutoff values in stepwise regression methods for simplifying the final regression model, and proposes a resampling procedure by introducing alternative estimates of boostedCross-validated PRESS values for deciding the number of observations to be omitted and number of folds/subsets subsequently in K-fold cross- validation.
Estimation of prediction error by using K-fold cross-validation
TLDR
This paper investigates two families that connect the training error and K-fold cross-validation, which has a downward bias and has an upward bias.
Consistent Cross Validation with stable learners
TLDR
A debiased version of the K-fold is proposed which is consistent for any uniformly stable learner and applies to the problem of model selection and demonstrates empirically the usefulness of the promoted approach on real world datasets.
A survey of cross-validation procedures for model selection
TLDR
This survey intends to relate the model selection performances of cross-validation procedures to the most recent advances of model selection theory, with a particular emphasis on distinguishing empirical statements from rigorous theoretical results.
On optimal data split for generalization estimation and model selection
  • J. Larsen, Cyril Goutte
  • Computer Science
    Neural Networks for Signal Processing IX: Proceedings of the 1999 IEEE Signal Processing Society Workshop (Cat. No.98TH8468)
  • 1999
TLDR
The theoretical basics of various cross-validation techniques are described with the purpose of reliably estimating the generalization error and optimizing the model structure for reliably estimating a single location parameter.
An empirical comparison of $$V$$V-fold penalisation and cross-validation for model selection in distribution-free regression
TLDR
Cases in which VFCV and $$V$$V-fold penalisation provide poor estimates of the risk, respectively, are highlighted, and a modified penalisation technique is introduced to reduce the estimation error.
...
1
2
3
4
5
...

References

SHOWING 1-10 OF 18 REFERENCES
An alternative method of cross-validation for the smoothing of density estimates
TLDR
An alternative method of cross-validation, based on integrated squared error, recently also proposed by Rudemo (1982), is derived, and Hall (1983) has established the consistency and asymptotic optimality of the new method.
The Predictive Sample Reuse Method with Applications
TLDR
A recently devised method of prediction based on sample reuse techniques that is most useful in low structure data paradigms that involve minimal assumptions is presented.
Classification and Regression Trees
TLDR
This chapter discusses tree classification in the context of medicine, where right Sized Trees and Honest Estimates are considered and Bayes Rules and Partitions are used as guides to optimal pruning.
Generalized $L-, M-$, and $R$-Statistics
Abstract : A class of statisticss generalizing U-statistics and L-statistics, and containing other varieties of statistics as well, such as trimmed U-statistics, is studied. Using the differentiable
Estimating Optimal Transformations for Multiple Regression and Correlation.
Abstract In regression analysis the response variable Y and the predictor variables X 1 …, Xp are often replaced by functions θ(Y) and O1(X 1), …, O p (Xp ). We discuss a procedure for estimating
Optimal Bandwidth Selection in Nonparametric Regression Function Estimation
On considere des estimateurs du noyau d'une fonction de regression multivariable et une regle de selection selon la largeur de bande formulee en terme de validation croisee
Jackknife Approximations to Bootstrap Estimates
Let T be an estimate of the form Tn = T(F ) where F is the nn n' n sample cdf of n iid observations and T is a locally quadratic functional defined on cdf's. Then, the normalized jackknife estimates
Approximation Theorems of Mathematical Statistics
Preliminary Tools and Foundations. The Basic Sample Statistics. Transformations of Given Statistics. Asymptotic Theory in Parametric Inference. U--Statistics. Von Mises Differentiable Statistical
Estimation of optimal transformations using D-fold cross validation and repeated learning-testing methods
  • Sankhya A 51. To appear. GEISSER,
  • 1989
Estimation of optimal transformations using D-fold cross validation and repeated learning-testing methods. Sankhya A 51
  • Estimation of optimal transformations using D-fold cross validation and repeated learning-testing methods. Sankhya A 51
  • 1989
...
1
2
...