Bayesian Model Assessment and Comparison Using Cross-Validation Predictive Densities
@article{Vehtari2002BayesianMA, title={Bayesian Model Assessment and Comparison Using Cross-Validation Predictive Densities}, author={Aki Vehtari and Jouko Lampinen}, journal={Neural Computation}, year={2002}, volume={14}, pages={2439-2468} }
In this work, we discuss practical methods for the assessment, comparison, and selection of complex hierarchical Bayesian models. A natural way to assess the goodness of the model is to estimate its future predictive capability by estimating expected utilities. Instead of just making a point estimate, it is important to obtain the distribution of the expected utility estimate because it describes the uncertainty in the estimate. The distributions of the expected utility estimates can also be…
197 Citations
Expected Utility Estimation via Cross-Validation
- Computer Science
- 2003
A quick and generic approach based on the Bayesian bootstrap for obtaining samples from the distributions of the expected utility estimates, which can also be used for model comparison by computing the probability of one model having a better expected utility than some other model.
Expected utility estimation via cross-validation
- Business
- 2003
SUMMARY We discuss practical methods for the assessment, comparison and selection of complex hierarchical Bayesian models. A natural way to assess the goodness of the model is to estimate its future…
Bayesian Leave-One-Out Cross-Validation Approximations for Gaussian Latent Variable Models
- Computer ScienceJ. Mach. Learn. Res.
- 2016
This article considers Gaussian latent variable models where the integration over the latent values is approximated using the Laplace method or expectation propagation and finds the approach based upon a Gaussian approximation to the LOO marginal distribution gives the most accurate and reliable results among the fast methods.
Practical Bayesian model evaluation using leave-one-out cross-validation and WAIC
- Computer ScienceStatistics and Computing
- 2016
An efficient computation of LOO is introduced using Pareto-smoothed importance sampling (PSIS), a new procedure for regularizing importance weights, and it is demonstrated that PSIS-LOO is more robust in the finite case with weak priors or influential observations.
Practical Bayesian model evaluation using leave-one-out cross-validation and WAIC
- Computer ScienceStat. Comput.
- 2017
An efficient computation of LOO is introduced using Pareto-smoothed importance sampling (PSIS), a new procedure for regularizing importance weights, and it is demonstrated that PSIS-LOO is more robust in the finite case with weak priors or influential observations.
Bayesian Input Variable Selection Using Posterior Probabilities and Expected Utilities
- Computer Science, Economics
- 2002
Benefits of using expected utilities for input variable selection in complex Bayesian hierarchical models are that it is less sensitive to prior choices and it provides useful model assessment, and it helps finding useful models.
Model selection via predictive explanatory power
- Computer Science
- 2004
This work proposes a model selection method based on Kullback-Leibler divergence from the predictive distribution of the full model to the predictive distributions of the submodels, and compares the performance of the method to posterior probabilities, deviance information criteria, and direct maximization of the expected utility via crossvalidation.
Erratum to: Practical Bayesian model evaluation using leave-one-out cross-validation and WAIC
- Computer ScienceStat. Comput.
- 2017
An efficient computation of LOO is introduced using Pareto-smoothed importance sampling (PSIS), a new procedure for regularizing importance weights, and it is demonstrated that PSIS-LOO is more robust in the finite case with weak priors or influential observations.
Uncertainty in Bayesian Leave-One-Out Cross-Validation Based Model Comparison
- Computer Science
- 2020
It is shown that it is possible that the problematic skewness of the error distribution, which occurs when the models make similar predictions, does not fade away when the data size grows to infinity in certain situations.
References
SHOWING 1-10 OF 77 REFERENCES
Bayesian model assessment and selection using expected utilities
- Computer Science, Economics
- 2001
This work proposes an approach using cross-validation predictive densities to compute the expected utilities and Bayesian bootstrap to obtain samples from their distributions, and proposes to use a variable dimension Markov chain Monte Carlo method to find out potentially useful input combinations.
On Bayesian Model Asessment and Choice Using Cross-Validation Predictive Densities
- Computer Science
- 2001
It is demonstrated that in flexible non-linear models having many parameters, the importance sampling approximated leave-one-out cross-validation (IS-LOO-CV) proposed in (Gelfand et al., 1992) may not work.
Posterior Predictive Assessment of Model Fitnessvia Realized
- Mathematics
- 1995
This paper considers the Bayesian counterparts of the classical tests for goodness of t and their use in judging the t of a single Bayesian model to the observed data. We focus on posterior…
Bayesian Input Variable Selection Using Cross-Validation Predictive Densities and Reversible Jump MCMC
- Computer Science
- 2001
This work proposes to use the reversible jump Markov chain Monte Carlo (RJMCMC) method to find out potentially useful input combinations, for which the final model choice and assessment is done using the cross-validation predictive densities, and discusses why the posterior probabilities of the models given by the RJMCMC should not be used directly for input selection.
POSTERIOR PREDICTIVE ASSESSMENT OF MODEL FITNESS VIA REALIZED DISCREPANCIES
- Mathematics
- 1996
This paper considers Bayesian counterparts of the classical tests for good- ness of fit and their use in judging the fit of a single Bayesian model to the observed data. We focus on posterior…
Approximate Bayesian-inference With the Weighted Likelihood Bootstrap
- Mathematics
- 1994
We introduce the weighted likelihood bootstrap (WLB) as a way to simulate approximately from a posterior distribution. This method is often easy to implement, requiring only an algorithm for…
On Bayesian Model Assessment and Choice Using Cross-Validation Predictive Densities: Appendix
- Computer Science
- 2001
The prior and the MCMC specification details are given here and the notation r ∼ F(a) as shorthand for p(r) = F(r |a) where a denotes the parameters of the distribution F , and the random variable argument r is not shown explicitly.
Bayesian approach for neural networks--review and case studies
- Computer ScienceNeural Networks
- 2001
Computing Bayes Factors by Combining Simulation and Asymptotic Approximations
- Mathematics
- 1997
Abstract The Bayes factor is a ratio of two posterior normalizing constants, which may be difficult to compute. We compare several methods of estimating Bayes factors when it is possible to simulate…
Bayesian Inference in Econometric Models Using Monte Carlo Integration
- Mathematics
- 1989
Methods for the systematic application of Monte Carlo integration with importance sampling to Bayesian inference are developed. Conditions under which the numerical approximation converges almost…