• Corpus ID: 14534062

Estimating out-of-sample pointwise predictive accuracy using posterior simulations

@inproceedings{Vehtari2014EstimatingOP,
  title={Estimating out-of-sample pointwise predictive accuracy using posterior simulations},
  author={Aki Vehtari and Andrew Gelman},
  year={2014}
}
The Watanabe-Akaike information criterion (WAIC) and cross-validation are methods for estimating pointwise out-of-sample prediction accuracy from a fitted Bayesian model. WAIC is based on the series expansion of leave-one-out cross-validation (LOO), and asymptotically they are equal. With finite data, WAIC and cross-validation address different predictive questions and thus it is useful to be able to compute both. WAIC and an importance-sampling approximated LOO can be estimated directly using… 

Figures and Tables from this paper

Boba: Authoring and Visualizing Multiverse Analyses
TLDR
Baba: an integrated domain-specific language (DSL) and visual analysis system for authoring and reviewing multiverse analyses and demonstrates Boba's utility through two data analysis case studies, and reflects on challenges and design opportunities for multiverse analysis software.
Infanticide by females is a leading source of juvenile mortality in a large social carnivore
TLDR
Of four hypotheses regarding the evolution of infanticide, the most support is found for the hypothesis thatinfanticide in spotted hyenas reflects competition over social status among matrilines.
Influence of warming on microbial ecosystems
Climate change is progressing fast causing losses in biodiversity to the extent that scientists believe the world to be on the brink of the sixth wave of mass extinction. While some global change
Soil nitrogen mediates the effect of climatic distance on herbivory rates in a tropical herb
The center‐periphery hypothesis predicts that species are most abundant at the center of their distribution range. Differential herbivory rates between center and periphery populations can explain
Interactive effects of shifting body size and feeding adaptation drive interaction strengths of protist predators under warming
TLDR
Functional response experiments on bacterial prey along an experimental temperature gradient found an increase of maximum feeding rates and half-saturation densities with rising experimental temperatures, and adaptation to different temperatures potentially avoids an energetic mismatch with warming.
Sub-Perception and Supra-Perception Spinal Cord Stimulation in Chronic Pain Syndrome: A Randomized, Semi-Double-Blind, Crossover, Placebo-Controlled Trial
TLDR
The pain-relieving effects of SCS reached significance and were comparable across all modes of stimulation including sham, and the modality most preferred by patients was low frequency (55% or 10 patients).

References

SHOWING 1-10 OF 23 REFERENCES
Bayesian Model Assessment and Comparison Using Cross-Validation Predictive Densities
TLDR
This work proposes an approach using cross-validation predictive densities to obtain expected utility estimates and Bayesian bootstrap to obtain samples from their distributions, and discusses the probabilistic assumptions made and properties of two practical cross- validate methods, importance sampling and k-fold cross- validation.
Understanding predictive information criteria for Bayesian models
TLDR
The Akaike, deviance, and Watanabe-Akaike information criteria are reviewed from a Bayesian perspective and it is better understood, through small examples, how these methods can apply in practice.
Case-deletion importance sampling estimators: Central limit theorems and related results
Case-deleted analysis is a popular method for evaluating the influence of a subset of cases on inference. The use of Monte Carlo estima- tion strategies in complicated Bayesian settings leads
Model Determination Using Predictive Distributions with Implementation via Sampling-Based Methods
TLDR
Model determination is divided into the issues of model adequacy and model selection and it is proposed to validate conditional predictive distributions arising from single point deletion against observed responses.
Predictive likelihood for Bayesian model selection and averaging
Sequential Imputations and Bayesian Missing Data Problems
TLDR
This article introduces an alternative procedure that involves imputing the missing data sequentially and computing appropriate importance sampling weights, and in many applications this new procedure works very well without the need for iterations.
Penalized loss functions for Bayesian model comparison.
TLDR
DIC is shown to be an approximation to a penalized loss function based on the deviance, with a penalty derived from a cross-validation argument, which under-penalizes more complex models.
Asymptotic Equivalence of Bayes Cross Validation and Widely Applicable Information Criterion in Singular Learning Theory
TLDR
The Bayes cross-validation loss is asymptotically equivalent to the widely applicable information criterion as a random variable and model selection and hyperparameter optimization using these two values are asymPTOTically equivalent.
A survey of Bayesian predictive methods for model assessment, selection and comparison
TLDR
A unified review of Bayesian predictive model assessment and selection methods, and of methods closely related to them, with an emphasis on how each method approximates the expected utility of using a Bayesian model for the purpose of predicting future data.
Model-Based Clustering of Non-Gaussian Panel Data Based on Skew-t Distributions
We propose a model-based method to cluster units within a panel. The underlying model is autoregressive and non-Gaussian, allowing for both skewness and fat tails, and the units are clustered
...
...