Bayesian Model Assessment and Comparison Using Cross-Validation Predictive Densities

@article{Vehtari2002BayesianMA,
  title={Bayesian Model Assessment and Comparison Using Cross-Validation Predictive Densities},
  author={Aki Vehtari and Jouko Lampinen},
  journal={Neural Computation},
  year={2002},
  volume={14},
  pages={2439-2468}
}
In this work, we discuss practical methods for the assessment, comparison, and selection of complex hierarchical Bayesian models. A natural way to assess the goodness of the model is to estimate its future predictive capability by estimating expected utilities. Instead of just making a point estimate, it is important to obtain the distribution of the expected utility estimate because it describes the uncertainty in the estimate. The distributions of the expected utility estimates can also be… 
View on MIT Press
becs.aalto.fi
192 Citations
Highly Influential Citations
9
Background Citations
48
Methods Citations
86
Results Citations
5

192 Citations

Expected Utility Estimation via Cross-Validation

A quick and generic approach based on the Bayesian bootstrap for obtaining samples from the distributions of the expected utility estimates, which can also be used for model comparison by computing the probability of one model having a better expected utility than some other model.

Expected utility estimation via cross-validation

SUMMARY We discuss practical methods for the assessment, comparison and selection of complex hierarchical Bayesian models. A natural way to assess the goodness of the model is to estimate its future…

Bayesian Leave-One-Out Cross-Validation Approximations for Gaussian Latent Variable Models

This article considers Gaussian latent variable models where the integration over the latent values is approximated using the Laplace method or expectation propagation and finds the approach based upon a Gaussian approximation to the LOO marginal distribution gives the most accurate and reliable results among the fast methods.

Practical Bayesian model evaluation using leave-one-out cross-validation and WAIC

An efficient computation of LOO is introduced using Pareto-smoothed importance sampling (PSIS), a new procedure for regularizing importance weights, and it is demonstrated that PSIS-LOO is more robust in the finite case with weak priors or influential observations.

Bayesian Input Variable Selection Using Posterior Probabilities and Expected Utilities

Benefits of using expected utilities for input variable selection in complex Bayesian hierarchical models are that it is less sensitive to prior choices and it provides useful model assessment, and it helps finding useful models.

Bayesian nonparametric model selection and model testing

Model selection via predictive explanatory power

This work proposes a model selection method based on Kullback-Leibler divergence from the predictive distribution of the full model to the predictive distributions of the submodels, and compares the performance of the method to posterior probabilities, deviance information criteria, and direct maximization of the expected utility via crossvalidation.

Erratum to: Practical Bayesian model evaluation using leave-one-out cross-validation and WAIC

An efficient computation of LOO is introduced using Pareto-smoothed importance sampling (PSIS), a new procedure for regularizing importance weights, and it is demonstrated that PSIS-LOO is more robust in the finite case with weak priors or influential observations.

Uncertainty in Bayesian Leave-One-Out Cross-Validation Based Model Comparison

It is shown that it is possible that the problematic skewness of the error distribution, which occurs when the models make similar predictions, does not fade away when the data size grows to infinity in certain situations.

Bayesian Input Variable Selection Using Cross-Validation Predictive Densities and Reversible Jump MCMC

This work proposes to use the reversible jump Markov chain Monte Carlo (RJMCMC) method to find out potentially useful input combinations, for which the final model choice and assessment is done using the cross-validation predictive densities, and discusses why the posterior probabilities of the models given by the RJMCMC should not be used directly for input selection.
...

References

SHOWING 1-10 OF 77 REFERENCES

Bayesian model assessment and selection using expected utilities

This work proposes an approach using cross-validation predictive densities to compute the expected utilities and Bayesian bootstrap to obtain samples from their distributions, and proposes to use a variable dimension Markov chain Monte Carlo method to find out potentially useful input combinations.

On Bayesian Model Asessment and Choice Using Cross-Validation Predictive Densities

It is demonstrated that in flexible non-linear models having many parameters, the importance sampling approximated leave-one-out cross-validation (IS-LOO-CV) proposed in (Gelfand et al., 1992) may not work.

Posterior Predictive Assessment of Model Fitnessvia Realized

This paper considers the Bayesian counterparts of the classical tests for goodness of t and their use in judging the t of a single Bayesian model to the observed data. We focus on posterior…

Bayesian Input Variable Selection Using Cross-Validation Predictive Densities and Reversible Jump MCMC

This work proposes to use the reversible jump Markov chain Monte Carlo (RJMCMC) method to find out potentially useful input combinations, for which the final model choice and assessment is done using the cross-validation predictive densities, and discusses why the posterior probabilities of the models given by the RJMCMC should not be used directly for input selection.

POSTERIOR PREDICTIVE ASSESSMENT OF MODEL FITNESS VIA REALIZED DISCREPANCIES

This paper considers Bayesian counterparts of the classical tests for good- ness of fit and their use in judging the fit of a single Bayesian model to the observed data. We focus on posterior…

Approximate Bayesian-inference With the Weighted Likelihood Bootstrap

We introduce the weighted likelihood bootstrap (WLB) as a way to simulate approximately from a posterior distribution. This method is often easy to implement, requiring only an algorithm for…

Markov Chain Monte Carlo Methods for Computing Bayes Factors

It is found that the joint model-parameter space search methods perform adequately but can be difficult to program and tune, whereas the marginal likelihood methods often are less troublesome and require less additional coding.

On Bayesian Model Assessment and Choice Using Cross-Validation Predictive Densities: Appendix

The prior and the MCMC specification details are given here and the notation r ∼ F(a) as shorthand for p(r) = F(r |a) where a denotes the parameters of the distribution F , and the random variable argument r is not shown explicitly.

Bayesian approach for neural networks--review and case studies

Computing Bayes Factors by Combining Simulation and Asymptotic Approximations

Abstract The Bayes factor is a ratio of two posterior normalizing constants, which may be difficult to compute. We compare several methods of estimating Bayes factors when it is possible to simulate…
...